Why Do Small Studies Disagree?

Why do small studies disagree?

Imagine reading that one study found a benefit from a new routine, then seeing another study report no benefit. It is tempting to ask which one is wrong. But studies do not observe an entire population. They observe particular people, on particular days, under particular conditions. Even when researchers follow the same sound procedure, the people they happen to include can lead to different results.

This is sampling variation: a statistic calculated from a sample changes from one sample to another. As OpenStax explains, variation is an expected feature of sampling. It matters especially when samples are small, because each observation has a relatively large influence on the reported result. Disagreement is therefore a reason to look more closely, not an automatic verdict that one team made a mistake.

A small, entirely synthetic example

Consider this entirely synthetic illustration, not a report of real research. Two independent studies each randomly assign an illustrative 20 people to a treatment and an illustrative 20 people to a comparison group. Improvement is recorded as a yes-or-no outcome. Assume, for this synthetic example, that the treatment has no real effect and that everyone has the same illustrative 50% chance of improving regardless of their group.

In synthetic Study A, an illustrative 12 of 20 treatment participants improve, compared with an illustrative 10 of 20 comparison participants. Those synthetic counts correspond to illustrative rates of 60% and 50%, an apparent treatment advantage of 10 percentage points. In synthetic Study B, an illustrative 9 of 20 treatment participants improve, compared with an illustrative 11 of 20 comparison participants. Those synthetic rates are 45% and 55%, an apparent treatment disadvantage of 10 percentage points.

The synthetic studies point in opposite directions despite sharing the same assumed underlying reality. Neither set of illustrative counts requires a change in the treatment’s true effect. Each study drew a different small collection of outcomes. This example does not show that real disagreements are always due to chance; it shows why opposite-looking results are possible even before differences in study design enter the picture.

Why each observation matters more

In the synthetic example, changing the outcome of one illustrative participant in a group of 20 changes that group’s reported improvement rate by 5 percentage points. A handful of different outcomes can therefore change the apparent direction of the comparison. In a larger group, any one participant contributes a smaller share of the total. That is the intuition behind why larger samples generally produce more stable estimates when the sampling process and underlying variability are otherwise comparable.

Stability does not mean every larger study will agree with every other one. Random variation remains, and outcomes that vary greatly between people can still be difficult to estimate precisely. The OpenStax discussion of sampling distributions describes how estimates fluctuate across repeated samples and how sample size affects that fluctuation for sample means. The exact uncertainty calculation depends on the outcome and study design, but the basic question carries across settings: how much might this estimate move if the study were repeated with different eligible participants?

A reported difference should therefore be read alongside its uncertainty, often expressed through a confidence interval. An interval is not a guarantee about the true value. It communicates a range produced by a stated statistical method, whose interpretation depends on the method’s assumptions. Two point estimates can look far apart while both remain too imprecise to establish a meaningful conflict.

Chance is not the only explanation

Before attributing disagreement to sampling variation, check whether the studies asked the same question. A routine might differ in duration, intensity, or timing. One study might recruit beginners while another recruits experienced participants. Researchers might define improvement differently or measure it after different follow-up periods. In those cases, the studies may be estimating genuinely different effects, so a direct comparison of their headline numbers can mislead.

Study methods also matter. Random assignment helps make treatment groups comparable within a study, while representative sampling concerns how well a study’s participants reflect a population of interest. Neither property follows automatically from having a large sample. A large study that systematically excludes relevant people can give a precise estimate for the wrong population. Likewise, missing outcomes, inconsistent measurements, or other biases can push results in a direction that more participants alone will not fix.

Finally, ask what was reported. If researchers measured many outcomes or examined several subgroups, an eye-catching result may be only part of the picture. A disagreement between published headlines can reflect different choices about which outcomes to emphasize. These possibilities do not establish misconduct or invalidate a study; they identify details needed to judge whether the findings address the same claim.

Reading apparently conflicting results

Start with the estimated effect, not just labels such as “worked” and “didn’t work.” Those labels can hide similar estimates on either side of a reporting threshold. Look for the outcome’s actual definition, the size and direction of each estimate, the number of participants contributing data, and the reported uncertainty. An estimate near no effect with wide uncertainty tells a different story from a similarly sized estimate measured with much greater precision.

Next, compare eligibility rules, assignment methods, interventions, follow-up periods, and missing-data handling. If these differ, write down which population and outcome each study can reasonably speak to. If they are closely matched, ask whether the observed gap is surprising given each study’s uncertainty. Overlapping confidence intervals can be a useful visual prompt, but overlap alone is not a formal test of whether study results differ.

The strongest answer often comes from considering the full body of comparable evidence rather than choosing a winner from two small reports. Replication helps show whether an apparent effect persists. A systematic review can examine how studies were selected and whether combining their results is appropriate. The goal is not to make every study yield the same number; it is to understand how much variation the evidence allows and which conclusions remain credible across studies.

A practical checklist

When two small studies seem to disagree, pause before treating either headline as the final answer. The disagreement may reflect ordinary sample-to-sample movement, different questions, or weaknesses in one or both designs. The first step is to separate those possibilities. A small result is not automatically wrong, and a large result is not automatically right.

Use this short checklist: What population and outcome did each study examine? Were participants selected and assigned comparably? How large are the estimated effects, and how uncertain are they? Could differences in timing, measurement, or missing data explain the gap? Do other comparable studies point consistently in either direction? If the answers remain unclear, the most accurate conclusion may be that the evidence is still uncertain.