The Fallacy of Large Numbers in Surveys

The fallacy of large numbers

A survey arrives with an impressive response count. Its result has barely changed as more replies came in, so it feels settled. The tempting conclusion is that a sample this large must reflect the population. That conclusion is the fallacy of large numbers: treating size alone as a guarantee that an estimate is accurate.

Large samples can be valuable. Under appropriate sampling conditions, they tend to vary less from sample to sample than small ones. But an estimate can be consistent and still miss its target. If some kinds of people are more likely to appear in a survey, collecting more responses through the same process can make the result more stable without making it more representative.

The question is therefore not just, “How many responses were there?” It is also, “How did these people get into the sample?” OpenStax distinguishes between a population and a sample and emphasizes that the way a sample is obtained matters when drawing conclusions about that population.

What a bigger sample actually improves

Imagine repeating a well-designed random sampling process and calculating a survey percentage each time. The percentages will differ because different people happen to be selected. This is sampling variation. With more observations, that random variation generally shrinks, provided the assumptions behind the sampling method are reasonable.

A related idea appears in the law of large numbers: under suitable conditions, an average of observations tends toward the expected value as the number of observations grows. This is a statement about the behavior of an average under a specified process. It does not say that every method of gathering a large pile of observations reveals the answer you intended to measure.

The distinction is the target of convergence. If the process reaches the population of interest fairly, more data can help estimate that population more closely. If the process repeatedly favors a particular group, more data can pin down the result produced by that process instead. Precision means an estimate is less variable; accuracy means it is close to the intended population value. A result can have one without the other.

A worked survey example

Entirely synthetic example: A city has an illustrative population of 100,000 households. An illustrative 50,000 would answer yes to a proposed service change, and an illustrative 50,000 would answer no. The population’s illustrative yes share is therefore 50%. These figures are invented solely to make the true answer visible for comparison.

Synthetic volunteer survey: Suppose an illustrative 7% of yes households and an illustrative 3% of no households respond to an online invitation. That produces an illustrative 3,500 yes responses and 1,500 no responses. The survey has an illustrative 5,000 responses, and its illustrative yes share is 70%. The arithmetic is correct, but the estimate is an illustrative 20 percentage points above the synthetic population value because willingness to respond differs by answer.

Synthetic random sample for comparison: Suppose a separate illustrative random sample of 400 households contains 210 yes answers. Its illustrative estimate is 52.5%, closer to the illustrative population value of 50% despite using fewer households. That particular small-sample result is not guaranteed: another random draw could land farther away. The comparison shows why a large response count cannot compensate automatically for unequal chances of entering the sample.

Why the error can survive growth

In the synthetic volunteer survey, the people who favor the change are more likely to appear. If the same invitation reaches more households while that imbalance persists, the response count can grow and the reported share can become steadier. Neither change ensures that the share moves toward the value for all city households. The missing ingredient is a selection process that supports the population claim.

This is selection bias, not simply bad luck. Random sampling variation reflects which eligible people happened to be drawn under a sampling design. Selection bias arises when the route into the sample systematically favors some answers or characteristics. OpenStax’s discussion of sampling methods explains why a convenience sample cannot be treated as equivalent to a random sample merely because it contains many observations.

The same concern appears beyond online polls. A customer feedback form can overrepresent people motivated to respond. A workplace survey can miss employees who have left. A health survey distributed only through a clinic does not automatically describe everyone in the surrounding community. In each case, ask whether the people absent from the data might differ in a way that matters to the question.

Read precise-looking claims carefully

A report may present a narrow interval or many decimal places beside a large-sample estimate. Those displays can communicate uncertainty from random sampling under a particular model. They do not necessarily include uncertainty caused by nonresponse, coverage gaps, misleading questions, or inaccurate answers. NIST’s statistical handbook treats study design and sources of variation as part of interpreting statistical results; a calculation cannot repair a mismatch between the data collected and the question asked.

This does not mean every opt-in survey is useless or every small random sample is trustworthy. A volunteer poll can accurately describe its respondents, and researchers may use additional information to adjust for known differences. A probability sample can still face nonresponse or measurement problems. The responsible claim depends on the sampling frame, response process, question wording, and assumptions behind any adjustment.

When reading a headline, translate “many people answered” into a narrower statement: many people who were reached and chose or were able to answer provided data. Whether that group supports a claim about a wider population requires evidence about selection, not just an impressive count.

A practical checklist

Before relying on a large-sample result, identify the population the claim is about and compare it with the people the data could actually reach. Find out whether respondents were randomly selected, recruited by convenience, or invited to volunteer. Check whether nonrespondents could plausibly answer differently and whether the report explains how it handled that risk.

Then read any uncertainty statement for what it covers. A small sampling error addresses only the uncertainty allowed for by its method and assumptions; it is not a certificate that the survey has no bias. Large numbers are useful when the data-generating process fits the question. When it does not, a bigger sample can give a more precise answer to the wrong question.