Sample Biases: Who Gets Left Out?

Sample biases begin with a missing voice

Imagine asking people whether their neighborhood needs a quieter street. You collect answers, calculate a percentage, and report what residents think. The calculation may be flawless, yet the conclusion may be misleading if the people who answered differ systematically from the people you meant to describe. That is the central problem behind sample biases.

A population is the full group you want to learn about; a sample is the group whose responses you actually observe. As OpenStax explains, a statistic calculated from a sample is used to learn about a population parameter. For that step to be credible, the route from population to sample matters as much as the arithmetic applied afterward.

Bias is not simply the possibility that a sample result differs from the population value. Samples also vary by chance. Sampling bias concerns a systematic problem in how people enter the sample or provide usable data. Collecting more responses through the same flawed process can make the reported percentage look more precise without fixing whom it represents.

Start with the group you mean

Before evaluating a survey, write down its target population in ordinary language. Does “the neighborhood” mean current residents, property owners, adults who live there, or everyone who travels through it? Each definition implies a different list of people to reach. A survey of property owners may answer a useful question, but it cannot automatically stand in for all residents.

Next ask about the sampling frame: the list or practical route used to contact people. If a neighborhood survey is distributed only through a homeowners’ email list, residents outside that list have no chance to receive it. This is a coverage problem. OpenStax’s discussion of sampling methods distinguishes methods that use random selection from convenience and voluntary-response approaches, where the way participants are found can limit what the results support.

Randomly choosing names from an incomplete list does not make the missing people appear. Random selection can reduce one source of selection bias among people on the list, but coverage still depends on who made it onto that list in the first place. The population definition and the frame should therefore be checked together.

How answers become unrepresentative

A convenience sample consists of people who are easy to reach. Asking visitors at a daytime community meeting might miss people who cannot attend then. A voluntary-response sample has a related issue: people decide for themselves whether to participate, and those with strong opinions may have more reason to respond. Neither method guarantees a particular direction of error; the concern is that inclusion is connected to the subject being measured.

Nonresponse can arise even when invitations go to a well-chosen group. Some selected people do not reply, and their views may differ from those of respondents. A reported response count tells you how many answers arrived, not whether the absent answers would have looked similar. Follow-up attempts and a clear account of who responded can help readers judge this risk, although they cannot reveal every missing opinion.

There is also a distinction worth keeping: a person can be included in the right sample but give an inaccurate answer because a question is confusing or sensitive. That is a response or measurement problem, not necessarily a problem with who was sampled. Both can distort a result, but they call for different remedies. Improving the contact list does not repair a leading question; rewriting the question does not reach people excluded by the contact list.

A worked example with synthetic data

Entirely synthetic illustration: Suppose a neighborhood has 200 adults, with one adult per household. The target question is whether all adults favor a proposed street change. An email survey reaches only 100 adults who belong to a community mailing list. Of those reached, 80 respond, and 60 of the respondents favor the change. Every count and percentage in this example is invented solely to show the calculation; none describes a real neighborhood.

In this synthetic survey, the reported support is 60 divided by 80, or 75% of respondents. That statement accurately describes the people who answered. It does not establish that 75% of all 200 adults favor the change: 100 adults were not on the email list, and another 20 received the survey but did not respond. The unobserved adults might have similar preferences, or they might not. The survey itself cannot settle that question.

For a synthetic comparison, imagine that a separate complete count found support among 90 of the 200 adults, or 45%. The synthetic email survey’s 75% would then overstate neighborhood support by 30 percentage points. The point is not that email surveys always overstate support, or that this gap is typical. It is that the respondent percentage and the target-population percentage describe different groups when coverage and response are uneven.

What a stronger survey changes

A better approach begins with a reachable frame for the stated population. Depending on the question, a survey team might draw addresses from a suitable list, use more than one contact method, and make repeated contact attempts. If participants are selected randomly, the team should explain the selection process and record how many selected people ultimately respond. These steps improve the basis for inference; they do not promise a perfectly unbiased result.

Researchers may also compare respondents with known population information and adjust weights when some groups are underrepresented. Weighting can help when the relevant differences are measured and the comparison information is sound. It cannot reliably recover the views of an entirely missing group merely by making the available respondents count more. A transparent report should say what was adjusted and what remains unknown.

For everyday reading, treat a precise-looking percentage as the end of a chain, not the start of the story. The chain runs from a defined population to a contact method, invitations, responses, and finally a calculation. A weakness early in that chain can matter more than another decimal place at the end.

A checklist for the next survey

When a result catches your attention, first identify its claim: is it about respondents, people contacted, or a broader population? Then look for a description of the sampling frame and selection method. If participation was voluntary, consider whether interest in the topic could be related to the answer. If many selected people did not respond, ask what is known about them. These questions do not automatically invalidate a survey; they establish how far its conclusion can reasonably travel.

Practical checklist: Define the target population; check who could be contacted; find out how people were selected; compare invitations with responses; read the exact question; and report remaining limits alongside the result. When those details are unavailable, describe the finding as a result among respondents, not as a settled fact about everyone.