A Bigger Biased Sample Is Still Biased

Ten thousand answers can miss the town

When a survey has ten thousand responses, the number itself feels authoritative. A hundred responses can feel flimsy by comparison. That intuition is useful only after asking how the respondents entered the sample. A large sample can estimate the characteristics of its own selection process with great precision while remaining systematically different from the target population.

Imagine a synthetic town of 100,000 adults. Exactly 60% would answer “satisfied” to a clearly defined service question. Half of the town belongs to group A, where 80% are satisfied. The other half belongs to group B, where 40% are satisfied. Those proportions are known because this is a programmed population, not a real survey.

Now put a voluntary response link in a place visited much more often by group A. If the resulting responses are 90% from A and 10% from B, the expected satisfaction rate among respondents is 76%, not the town’s 60%. Collecting more responses will make the result settle more firmly near 76%.

Two mechanisms at the same sample size

The sampling-bias experiment compares two mechanisms at the same selected sample size. The first draws each answer from the full synthetic town. The second draws from a tilted voluntary-response pool in which group A supplies 90% of the available answers. Run both methods repeatedly with one fixed seed that the reader can change.

At the smaller settings, the probability samples bounce around the 60% population value. Some runs land below it and others above it. The voluntary samples bounce around a different center, approximately 76%, because their group mix differs from the town. The paired histograms use the same number of answers and repetitions, making the change in center visible rather than confounding it with a different sample size.

Increase the shared sample-size slider from 50 toward 500. Both clouds narrow, one around 60% and the other around 76%. Then reduce group A’s share of the voluntary pool from 90% to 50%. Its center returns to the population value because the synthetic response mix now matches the town. This makes the distinction between variance and bias visible: low variability is not the same as accuracy.

All people, groups, response rates, and outcomes here are synthetic. The experiment supports a mathematical point about a specified mechanism. It makes no claim about an actual town, demographic group, public service, or survey vendor.

Separate center from spread

For an estimator theta_hat, statistical bias is:

Bias(theta_hat) = E(theta_hat) - theta

The expectation is taken over repetitions of the stated sampling process. In the probability design, every resident has the same inclusion probability, so the unweighted sample proportion has expectation 0.60. In the voluntary design, selection changes the group weights. Its expected result is:

0.90 * 0.80 + 0.10 * 0.40 = 0.76

Under an independent simple random sample, an estimated proportion has an approximate standard error:

SE(p_hat) = sqrt(p * (1 - p) / n)

The square root of n is the key. Multiplying the sample size by 100 divides this standard error by 10. The formula says nothing about the 16-percentage-point selection bias in the voluntary mechanism. As n grows, the random error approaches zero while the bias remains.

The phrase margin of error usually summarizes sampling variation under a probability design or a model that supplies equivalent assumptions. It does not include every source of error. Coverage gaps, nonresponse, measurement wording, duplicate responses, and processing errors can all remain outside a simple margin.

The UC Berkeley sampling chapter separates sampling error from selection and nonresponse problems. The two programmed mechanisms above make that separation observable because the target rate and both response mixtures are fixed in advance.

Fill the precision-versus-bias record

Keep the synthetic town rate at 60% and the voluntary pool at 90% group A and 10% group B, whose expected rate is 76%. For each proposed sample size, predict the center of the probability samples, calculate sqrt(0.60 * 0.40 / n), and record whether a larger n removes the voluntary sample’s 16-point distance from the town.

Answer. The probability-sample center remains 60% at every row. Its approximate standard error falls from 4.90 percentage points at n = 100, to 2.45 points at n = 400, to 0.49 points at n = 10,000. The voluntary-sample center remains 76%, so its distance from the programmed town remains 16 points. A large voluntary sample can therefore be very stable around the wrong target. The standard-error calculation describes the stated independent probability mechanism; it is not a margin of error for the voluntary design.

Conditions behind the simple comparison

The simple formula assumes independent draws and a defensible target population. Real probability samples can use stratification, clustering, unequal selection probabilities, and weights; their uncertainty must account for those design features. Sampling without replacement from a small finite population also changes the variance.

The example fixes response propensities for clarity. In practice, response behavior can vary across many observed and unobserved characteristics. Weighting may reduce bias when relevant population totals are known and the adjustment variables explain response and outcome differences. Weighting is not a magic repair when the needed information is absent.

Question wording is another mechanism. A perfectly selected sample can still produce a biased measurement if the question is leading, ambiguous, or interpreted differently across groups.

Precision claims that do not follow

Do not compare a small probability sample with a large opt-in sample and conclude that the former must be better in every respect. A poorly implemented probability design can suffer serious nonresponse or frame errors, while a carefully modeled nonprobability design may perform well for a particular target. The correct question is which assumptions are supported and how uncertainty was assessed.

Do not attach the simple random-sample margin of error to convenience data. The impressive-looking interval would describe random variation under a selection process that did not occur. It can be narrow and still miss the target entirely.

Do not use demographic resemblance alone as proof of representativeness. A sample can match the target on age and region while differing on unmeasured factors related to the outcome. Also avoid treating a response rate as a direct numerical measure of bias. A low rate raises concern, but the direction and size of bias depend on how respondents differ from nonrespondents.

How the sources bound the example

The U.S. Census Bureau explains why unit nonresponse matters and how differences between respondents and nonrespondents can bias estimates. The UC Berkeley statistics text separates selection, nonresponse, and sampling error and explains the role of probability sampling. This guide applies those principles to a new programmed population so every true value and response mechanism can be inspected.

Revision note, August 26, 2026. Reworked the headings, added the three-row precision-versus-bias record with a checked answer, and linked the Berkeley sampling discussion beside the distinction it supports. All town, group, response, and outcome values remain synthetic.