The misconception
When a survey has ten thousand responses, the number itself feels authoritative. A hundred responses can feel flimsy by comparison. That intuition is useful only after asking how the respondents entered the sample. A large sample can estimate the characteristics of its own selection process with great precision while remaining systematically different from the target population.
Imagine a synthetic town of 100,000 adults. Exactly 60% would answer “satisfied” to a clearly defined service question. Half of the town belongs to group A, where 80% are satisfied. The other half belongs to group B, where 40% are satisfied. Those proportions are known because this is a programmed population, not a real survey.
Now put a voluntary response link in a place visited much more often by group A. If the resulting responses are 90% from A and 10% from B, the expected satisfaction rate among respondents is 76%, not the town’s 60%. Collecting more responses will make the result settle more firmly near 76%.
A synthetic experiment
The sampling-bias experiment compares two mechanisms at the same selected sample size. The first draws each answer from the full synthetic town. The second draws from a tilted voluntary-response pool in which group A supplies 90% of the available answers. Run both methods repeatedly with one fixed seed that the reader can change.
At the smaller settings, the probability samples bounce around the 60% population value. Some runs land below it and others above it. The voluntary samples bounce around a different center, approximately 76%, because their group mix differs from the town. The paired histograms use the same number of answers and repetitions, making the change in center visible rather than confounding it with a different sample size.
Increase the shared sample-size slider from 50 toward 500. Both clouds narrow, one around 60% and the other around 76%. Then reduce group A’s share of the voluntary pool from 90% to 50%. Its center returns to the population value because the synthetic response mix now matches the town. This makes the distinction between variance and bias visible: low variability is not the same as accuracy.
All people, groups, response rates, and outcomes here are synthetic. The experiment supports a mathematical point about a specified mechanism. It makes no claim about an actual town, demographic group, public service, or survey vendor.
Derivation and reasoning
For an estimator theta_hat, statistical bias is:
Bias(theta_hat) = E(theta_hat) - theta
The expectation is taken over repetitions of the stated sampling process. In the probability design, every resident has the same inclusion probability, so the unweighted sample proportion has expectation 0.60. In the voluntary design, selection changes the group weights. Its expected result is:
0.90 * 0.80 + 0.10 * 0.40 = 0.76
Under an independent simple random sample, an estimated proportion has an approximate standard error:
SE(p_hat) = sqrt(p * (1 - p) / n)
The square root of n is the key. Multiplying the sample size by 100 divides this standard error by 10. The formula says nothing about the 16-percentage-point selection bias in the voluntary mechanism. As n grows, the random error approaches zero while the bias remains.
The phrase margin of error usually summarizes sampling variation under a probability design or a model that supplies equivalent assumptions. It does not include every source of error. Coverage gaps, nonresponse, measurement wording, duplicate responses, and processing errors can all remain outside a simple margin.
Assumptions
The simple formula assumes independent draws and a defensible target population. Real probability samples can use stratification, clustering, unequal selection probabilities, and weights; their uncertainty must account for those design features. Sampling without replacement from a small finite population also changes the variance.
The example fixes response propensities for clarity. In practice, response behavior can vary across many observed and unobserved characteristics. Weighting may reduce bias when relevant population totals are known and the adjustment variables explain response and outcome differences. Weighting is not a magic repair when the needed information is absent.
Question wording is another mechanism. A perfectly selected sample can still produce a biased measurement if the question is leading, ambiguous, or interpreted differently across groups.
Failure cases
Do not compare a small probability sample with a large opt-in sample and conclude that the former must be better in every respect. A poorly implemented probability design can suffer serious nonresponse or frame errors, while a carefully modeled nonprobability design may perform well for a particular target. The correct question is which assumptions are supported and how uncertainty was assessed.
Do not attach the simple random-sample margin of error to convenience data. The impressive-looking interval would describe random variation under a selection process that did not occur. It can be narrow and still miss the target entirely.
Do not use demographic resemblance alone as proof of representativeness. A sample can match the target on age and region while differing on unmeasured factors related to the outcome. Also avoid treating a response rate as a direct numerical measure of bias. A low rate raises concern, but the direction and size of bias depend on how respondents differ from nonrespondents.
Source notes
The U.S. Census Bureau explains why unit nonresponse matters and how differences between respondents and nonrespondents can bias estimates. The UC Berkeley statistics text separates selection, nonresponse, and sampling error and explains the role of probability sampling. This guide applies those principles to a new programmed population so every true value and response mechanism can be inspected.
Keep these distinctions
- Sample size reduces random sampling variation, not systematic selection error.
- A margin of error requires a defensible sampling or modeling process.
- Define the target population, sampling frame, and response mechanism before interpreting precision.
Sources checked
These sources support the statistical definitions and boundaries. The scenario, prose, calculations, and figure on this page are original to Count & Chance.
- Response Rates Definitions, U.S. Census Bureau. Checked 2026-08-19.
- Sampling, University of California, Berkeley Department of Statistics. Checked 2026-08-19.