Jewish World Review
The Media's Margin for Error
by Howard Fienberg and David Murray
SINCE PUBLIC OPINION POLLS have come to dominate news coverage of electoral politics, it comes as little surprise that media have had some difficulties disentangling themselves from that mind-set in this post-presidential election period. In the scramble, some media had difficulty distinguishing between voting and polling.
For instance, The New York Times declared "Surprise! Elections Have Margins of Error, Too," (Nov. 19) and later reminisced in an editorial that "Americans awoke on Nov. 8 to the realization that elections, like polls, have a margin of error" ("Updating the Way We Vote," Nov. 24). Fortunately, more and more journalists are including reference to the 'margin or error' when discussing opinion polls in order to emphasize their imperfections. Unfortunately, as these examples show, neither journalists nor the public seems to understand what that margin means.
'Margin of error' usually means the quantifiable sampling error built into a well-designed and executed public opinion poll. It is best to remember that there are two different types of error which often get conflated. The first is measurement error. We know it happens for a variety of reasons, from using broken thermometers to simple mistakes in addition. Such measurement error does indeed contribute to the "margins of error" of polls, and is the kind of error we find in counting votes. But while we know that measurement error is present, we are unable to determine the exact difference that it makes since its sources are unpredictable.
Sampling error is the other problem that gets referred to under the heading "margins of error." The source of error in this case is not a "mistake" as it is with measurement error. It appears necessarily every time we use a small sample to represent a large population. While sampling error plagues all surveys, it has one virtue - it can be specified beforehand, since it follows from inherent properties of probability theory. Polls, like elections, have numerous sources of error. But sampling error is the only margin that can actually be measured, and elections do not yet involve statistical sampling.
Finally, many journalists sought high and low for analogies to convey the closeness of the election outcome in Florida. One popular choice was that used by Prof. Lawrence Tribe when arguing before the US Supreme Court. Tribe compared the election to a "photo-finish," where we are looking again at the photo to see how it actually came out.
Pursuing this analogy further, however, gives us an interesting lesson. Whenever we magnify a photographer to get a closer look at the detail, we finally hit a point of diminishing returns. At some point, the effort to get a larger and larger blow-up of the finish line to see what actually happens begins to disappoint us. The image itself becomes increasingly blurry at high orders of magnification. That is, the more detail we can see, the more we lose focus, and can no longer find the distinct boundary lines that convey the information we are seeking. At some point the variation in the individual pixels blurs the line that marks the finish. We have gone too far down in a quest for resolution (in the photographic and the electoral sense) when the "noise" in the image exceeds the magnitude of the difference we are seeking.
Howard Fienberg is research analyst and David Murray is senior analyst with the Statistical Assessment Service (STATS), a nonprofit nonpartisan organization dedicated to improving public understanding of scientific and social research
return to Howard Fienberg's page