How do multiple comparisons affect results?

How do multiple comparisons affect results?

Suppose a team changes a website’s checkout page and checks whether the change affects purchases. It also checks clicks, time on page, newsletter sign-ups, and several customer groups. Each question may be reasonable. Together, though, they create more opportunities for an unusual result to appear by chance.

That is the multiple-comparisons problem: the interpretation of one promising test depends partly on how many other chances you gave yourself to find a promising test. It matters when comparing several treatments, examining many outcomes, or searching through subgroups after seeing the data. The issue is not that running more tests is wrong. It is that a result selected from many tests needs more context than a result from one planned test.

A p-value summarizes how surprising the observed data, or data more extreme, would be under a specified null hypothesis and statistical model. It is not the probability that the finding is false. Nor does an ordinary p-value for one test tell a reader how many other tests were attempted. OpenStax explains the role of null and alternative hypotheses; NIST discusses the added considerations when making multiple comparisons.

Why one small p-value can be misleading

A significance threshold sets a rule for deciding when a test result counts as evidence against its null hypothesis. If the null is true and a test’s assumptions hold, a threshold of 0.05 gives that test a 5% false-positive probability. This describes a single test over repeated uses of the procedure, not a guarantee about any particular result.

Now consider several tests whose null hypotheses are all true. Each offers another opportunity to cross the threshold. If the tests are independent, the probability that none crosses it is the product of their individual probabilities of not crossing it. Subtracting that probability from one gives the chance that at least one test crosses the threshold.

Real analyses often involve related outcomes or groups, so their tests may not be independent. The simple multiplication calculation then may not apply. The broader point remains: looking across many results and highlighting only the smallest p-value can make a chance fluctuation look like a discovery. Readers need to know what was tested, not just what was selected for display.

A worked example with synthetic results

Everything in this example is synthetic and illustrative. Imagine a hypothetical website team tests 20 independent outcomes after a checkout change. Assume, for this calculation only, that the change truly affects none of them, each test is valid, and each uses a prespecified 0.05 significance threshold. The probability that at least one test falls below the threshold is 1 − (1 − 0.05)^20, or about 0.642. Thus this synthetic setup would produce at least one nominally significant result in about 64.2% of repeated runs, despite having no real effects.

Suppose the smallest p-value in this synthetic set is 0.03. Reported alone, it passes the individual test’s 0.05 threshold. But it was selected from 20 opportunities. The calculation above does not tell us whether this particular finding is false; it shows why the selection process changes how compelling that 0.03 looks.

One possible adjustment is Bonferroni: divide the desired overall threshold by the number of comparisons. For these synthetic 20 tests, 0.05 ÷ 20 = 0.0025. The synthetic p-value of 0.03 does not meet that threshold. A different synthetic p-value of 0.001 would meet it. Bonferroni controls the probability of at least one false rejection across the specified family of tests, even when the tests are dependent, provided the individual tests are valid.

Decide which comparisons belong together

An adjustment needs a defined family of comparisons: the set of tests whose combined false-positive risk you intend to control. In the synthetic website example, all outcomes searched for a headline finding might belong to one family. If a primary outcome was chosen before collecting data and secondary outcomes were explicitly exploratory, reporting those roles separately helps readers understand what was planned.

There is no universal rule that every calculation in a project must share one adjustment. The choice depends on the question and on what claim the analysis will make. Comparing every version of a page with every other version is different from testing one prespecified comparison. Changing the family only after seeing which choice produces a favorable result defeats the purpose of the adjustment.

It also helps to distinguish planned tests from exploration. Exploring groups and outcomes can reveal useful questions for future work. But a pattern found by searching the current data deserves cautious language and, where feasible, a new test on data not used to choose it. Calling an exploratory result exploratory preserves its value without giving it the evidential status of a prespecified finding.

Adjustments involve a tradeoff

Bonferroni is straightforward, but a stricter threshold can make real effects harder to detect. Other multiple-comparison procedures can be less conservative in some settings. The right choice depends on the comparisons, their dependencies, and the kind of error the analysis aims to limit. NIST’s discussion of multiple comparisons provides context for selecting a method rather than treating one adjustment as universal.

An adjustment does not repair a poorly designed study, invalid test assumptions, biased measurement, or an outcome chosen because it looked favorable. Nor does failing to pass an adjusted threshold prove that an effect is absent. Statistical evidence is more informative alongside an estimated effect, its uncertainty, the study design, and the complete list of relevant comparisons.

For readers, the useful question is not simply, “Was any p-value below the threshold?” Ask what the investigators planned to test, how many comparisons they examined, and whether the reported claim accounts for that search. These details determine whether a striking number is a focused result or one standout among many possibilities.

A practical checklist

Before testing, name the main question, the primary outcome, and the comparisons that support the intended claim. Decide which results are confirmatory and which are exploratory. Choose an error-control approach appropriate to that plan, and document any changes made after seeing the data.

When reporting results, show the full relevant set of comparisons, the method used to address multiplicity, and effect estimates with uncertainty where appropriate. Describe a selected subgroup or outcome as exploratory when it was found through searching. A fresh study can then test whether the pattern holds.

For a quick reading check, ask: What was planned? What else was tested? Was the highlighted result adjusted or clearly labeled exploratory? Multiple comparisons do not make data useless. They change how much weight one promising result can bear.