Biases in Sampling: Who Gets Counted?

Biases in Sampling Start With the Target Population

A statistical sample is useful only in relation to a clearly defined population. The population might be all residents of a city, all customers who used a service during a period, or all students enrolled at a university. The target population is the group a researcher wants to describe or understand. A sample is the smaller group from which data are actually collected.

Biases in sampling occur when the process of selecting or measuring observations systematically favors some members of the target population over others. The result can be a sample that differs from the population in ways that matter to the question. This is different from ordinary sampling variability. Random samples can differ from one another by chance, while sampling bias tends to push results in a particular direction.

A precise research question therefore begins with more than a sample size. It asks who should be represented, how those people can be reached, who is likely to respond, and whether the questions measure the intended characteristic. NIST and university statistics courses treat these design decisions as central parts of data collection.

Coverage Bias: Who Is Missing?

Coverage bias, also called undercoverage, appears when the sampling frame does not adequately include the target population. A sampling frame is the practical list or system used to select potential participants. If the frame leaves out a group, random selection from that frame cannot restore the missing group. Randomness protects selection among listed units; it does not correct an incomplete list.

For example, a survey about local transportation that recruits only through a smartphone application may miss people without access to that application. A survey of workplace conditions sent only to current employees may miss former employees whose experiences are relevant. These examples do not prove that the results are biased in a particular direction, because the direction depends on how the excluded group differs from the rest. They do show why coverage must be evaluated before interpreting an estimate.

Researchers can reduce coverage problems by combining contact methods, maintaining an appropriate sampling frame, using probability sampling when feasible, and comparing the achieved sample with known population information. Weighting can sometimes adjust for unequal representation, but it depends on reasonable information about the population and does not automatically repair every design problem.

Nonresponse and Voluntary Response Bias

Nonresponse bias occurs when selected people do not provide usable data and the people who respond differ meaningfully from those who do not. A high response rate is helpful, but it is not a guarantee of an unbiased result. A lower response rate is a warning that deserves investigation, not proof that the result is unusable. The key issue is whether response is related to the topic being measured.

Voluntary response sampling is a related risk. In a voluntary response survey, people decide for themselves whether to participate, often after seeing an open invitation. People with strong opinions, unusual experiences, or more available time may be especially motivated to answer. Their responses can be informative, but they should not automatically be treated as representative of everyone in the target population.

Follow-up contacts, accessible survey formats, translated materials, incentives, and multiple modes of contact may improve participation. Researchers should also report how participants were recruited, how many were contacted, and how many supplied usable responses. Transparent reporting lets readers judge the possibility of nonresponse bias rather than relying on the sample size alone.

Convenience Sampling and Selection Effects

Convenience sampling selects people who are easiest to reach. It is common in classroom exercises, pilot studies, product testing, and early-stage research because it can be quick and inexpensive. Convenience data may help researchers discover possible patterns or refine a questionnaire, but convenience alone does not justify generalizing to a wider population.

Selection bias can also arise when participation depends on an outcome or condition related to the research question. A study of customer satisfaction that includes only customers who completed a support interaction may omit people who abandoned the process. A study of successful projects that excludes projects that were canceled cannot describe the full experience of project work. This kind of missingness can make outcomes look better or more consistent than they are.

The practical question is always comparative: who had a realistic chance of entering the sample, and who did not? Researchers should describe eligibility rules, recruitment channels, exclusions, and the timing of data collection. Readers should be cautious when a conclusion about a broad population is based on a narrow or self-selected group.

Worked Example: A Synthetic Estimate

The following worked example uses entirely synthetic and illustrative numbers. Suppose a fictional organization wants to estimate the share of all its members who use a particular service. The target population contains two groups: members who usually use the organization’s website and members who usually use its phone service. The organization sends a survey only by email, and the phone-service group is less likely to respond.

For illustration, imagine a synthetic population of one thousand members, divided equally between the two groups. Suppose the website group has an illustrative service-use rate of sixty percent, while the phone-service group has an illustrative rate of twenty percent. The true population rate in this synthetic setup is therefore forty percent.

Now suppose the email survey receives synthetic responses from one hundred website-oriented members and twenty phone-service-oriented members. Those response counts are illustrative only. If the response rates within the groups match the synthetic service-use rates, the survey contains sixty service users from the website group and four from the phone-service group. The combined sample estimate is sixty-four service users out of one hundred twenty respondents, or about fifty-three percent.

The estimate is higher than the synthetic population rate because the group with greater service use is overrepresented among respondents. A larger email survey would reduce random variation within the email respondents, but it would not necessarily remove this nonresponse bias. To improve the design, the organization could contact members through more than one channel, follow up with sampled nonrespondents, or use a sampling plan that gives both groups a known and appropriate chance of selection.

A Practical Checklist for Better Samples

Before trusting a sample-based conclusion, identify the target population and ask whether the sampling frame covers it. Check how people were selected, whether participation was voluntary, and whether any groups were difficult to reach. Examine who did not respond and whether the topic could affect willingness to participate.

Also inspect the wording and mode of measurement. Leading or confusing questions can create measurement bias even when the sample was selected well. Compare the sample with credible population information when such information exists, and distinguish a probability sample from a convenience sample in the final report.

Practical checklist:

• Define the target population before collecting data.
• Check whether the sampling frame leaves out relevant groups.
• Record recruitment, response, follow-up, and exclusion procedures.
• Look for differences between respondents and nonrespondents.
• Treat convenience and voluntary response samples as limited evidence.
• State clearly which population the results can reasonably describe.