Internet Polls: Why Size Doesn't Matter

by Howard Fienberg
Janury 10, 2001

When the BBC pronounced that 'Britons are miserable - it's official' (1), I could imagine you muttering, 'So? We already knew that'. But wait a second.

Sure, you might be a morose self-deprecating blob, but how can we know that everybody else is the same? We can take a poll.

This particular poll was taken over the internet by Netdoctor.co.uk on 16 October 2000. Of the 400 respondents to the 'Emotional health survey', nearly one quarter feared a 'hopeless future', one in three felt 'downright miserable', and nearly one in 10 thought 'their death would make things better for others'. Unfortunately, the poll was about as valid as consulting a crystal ball.

There are obvious advantages to conducting polls online. A telephone poll may take several days and cost thousands of dollars, but an online poll can be ready overnight and for a fraction of the cost. Online polls regularly capture large numbers of responses, which might seem to make them more accurate than their less hi-tech alternatives. For instance, across the pond, an ABC News poll on internet addiction in the spring of 1998 yielded 17,251 responses - 17 times the sample of a typical telephone poll.

Sounds impressive - but in this case, size doesn't matter. Telephone polls randomly select participants, but online poll respondents take the initiative to make their views known. Hence, online polls capture the opinions of a group that is self-selected and does not represent the general population. Even worse, respondents can often skew the results by voting repeatedly. In May 2000, a poll by Canada's Toronto Star on a political debate supposedly was programmed to allow people to vote only once. But hackers generated more than 9500 votes for one political party from a single computer. Nine thousand votes for a second party came from another computer.

There are also significant demographic differences between the respondents to internet and telephone polls. This is to be expected, because there are social distinctions between those who have the time and inclination to accept a telephone poll from a stranger and those who surf the net. Less than half of Britons use the internet. These users are relatively better educated, more affluent and younger than the general population. Because pollsters cannot produce random email addresses the way they can do with telephone numbers, random sampling online is impossible. Still, proponents of internet polling think that statistical weighting techniques can be used to adjust results and produce representative samples.

Once computers with internet access become as common as the telephone, demographic deficiencies will dissipate and techniques of statistical weighting may be workable. Although security problems will never disappear, the ability to tag respondents with unique identifiers is within reach. Some day, a good online poll should be able to yield a number of quality responses from a reliable sample that far surpasses any traditional poll. But that day has yet to arrive. As long as the wired population is limited and unclear, so are the results of any wired survey.

Agony aunt Christine Webber, who carried out the internet survey for Netdoctor.co.uk, said: 'It seems people's lives do not live up to their extremely high expectations.' A quarter of her respondents said that life was unfair. You don't need a survey to tell you that. But aren't high expectations necessary, sometimes? For instance, the Beeb might have demanded a scientific poll before declaring the country a moping mob - a caricature which might be just as unfair as life.

Howard Fienberg is research analyst with the non-profit, non-partisan Statistical Assessment Service (STATS), a Washington DC-based organisation dedicated to bettering public understanding of scientific and social research

[see the original at Spiked]

return to Howard Fienberg's page