The Fort Worth Star-Telegram
By the Numbers: Online polling is not yet trustworthy
by Howard Fienberg, special to the Star-Telegram
December 12, 1999

The growth of the Internet, combined with our society's obsession with opinion polls, was bound to spawn a simpler way to administer more polls. So, enter online polling. Gordon S. Black, CEO of Harris Interactive, intends to lead the way and has committed $ 55 million to Internet polling for upcoming American political elections and referendums.

Columnist William Safire once said, "Sampling is no science. " He was and remains wrong. However, I can safely say that, for now, Internet polling is no science. And Harris' results will be worthless.

A telephone poll may take several days and thousands of dollars to set up, but an online poll can be ready overnight and for a fraction of the cost. Online polls also regularly capture large numbers of responses, which would seem to make online polls more accurate. For instance, an ABC News poll on Internet addiction this spring yielded 17,251 responses - 17 times the normal sample of a telephone poll.

But in this case, size doesn't matter. Telephone polls randomly select participants, but online poll respondents take the initiative to make their views known. Hence, online polls capture the opinions of a group that is self-selected and does not represent the general population.

Even worse, respondents often can vote repeatedly, skewing the data. A May poll by The Toronto Star on a political debate supposedly was programmed to allow people to vote only once, but hackers generated more than 9,500 votes for one political party from a single computer. Another 9,000 votes for a second party came from one other computer.

There are also significant demographic differences between the respondents of Internet and phone polls. This is to be expected, because there are social distinctions between those who have the time and inclination to accept a telephone poll from a stranger and those who command the Internet.

Online polls are limited to the roughly 40 percent of Americans using the Internet. According to the Pew Research Center, registered voters who go online are relatively well-educated and affluent. They are also young, while most studies indicate that the over-65 crowd votes more than any other demographic.

College graduates are substantially over-represented - 42 percent of those online, compared to 25 percent of all registered voters - as are members of families with incomes of more than $ 50,000 a year. In the center's own online poll, political independents participated in disproportionate numbers.

To explore how such differences affect results, the Pew Research Center conducted two simultaneous polls this spring - one by telephone, the other online. The results were vastly different on some key issues. For example, online poll participants "pay closer attention to election news" and were "more supportive of [President] Clinton's impeachment."

Despite the risks, Harris Interactive has tried to improve results of traditional methods by having millions of "cooperative respondents" join a database and by inviting selected people to respond to each survey, based on the demographics that they submit.

It then will adjust results for demographic discrepancies between the real world and the wired one.

Although it uses a more rigorous method than the average Internet poll, Harris' poll still suffers from a self-selected sample. As Warren J. Mitofsky wrote in a summer issue of Public Perspective, "if you want to survey people who do not have computers or who are not online, you cannot do that online. " No amount of statistical manipulation will change that.

There are certain advantages to online polling. Once computers with Internet access become as common as the telephone, demographic deficiencies will dissipate and techniques of statistical weighting will be made workable. Although security problems will never go away, the ability to tag respondents with unique identifiers is within reach. Someday a good online poll should be able to yield a number of quality responses from a reliable small sample that far surpasses any traditional poll.

But that day has not yet arrived. According to Professor Alan J. Rosenblatt of George Mason University, "the numbers suggest that the rate of usage is growing slowly and that it may take a while for the last third of the population to join the ranks of computer users. " As long as our wired population is limited, so are the results of any survey.

Pollsters have come a long way toward accuracy from the "Dewey Defeats Truman" headline days, and it would be unfortunate to discard that progress in pursuit of the latest fashion. To prove it to themselves, maybe pollsters should ask their clients a question: "Do you want opinion polling to be (1.) technologically flashy or (2.) accurate? "

I think I know what that poll would say.

Howard Fienberg is a research analyst with the Statistical Assessment Service, a nonprofit, nonpartisan organization based in Washington and working with the media to improve public understanding of science and statistics.

return to Howard Fienberg's page