Twenty Tests, One Discovery

The misconception

A result with p < 0.05 is often translated into one of three incorrect statements: there is less than a 5% chance the null hypothesis is true, there is less than a 5% chance the result happened randomly, or the effect has a 95% chance of being real. A p-value does not provide any of those probabilities.

It measures how incompatible the observed test statistic is with a specified statistical model, usually including a null hypothesis and assumptions about data generation. It is a conditional probability of data at least as extreme as what was observed, calculated as if that null model were true.

Even that interpretation is incomplete when many analyses were attempted. A threshold calibrated for one planned test does not automatically control the chance of a false claim across twenty opportunities.

A synthetic experiment

Generate twenty independent valid p-values under true null hypotheses. Under a continuous valid null model, each p-value has a uniform distribution between zero and one. This direct generator removes subject-matter distractions and isolates the selection problem. Apply alpha = 0.05 to every outcome and highlight p-values below 0.05.

There are no real participants, products, interventions, or outcomes. The generator sets every null effect to exactly zero. Nevertheless, some batches of twenty show one or more highlighted results. Change the seed and the location of the smallest p-value changes. Repeat the entire twenty-test batch thousands of times and count how often at least one false positive appears.

The linked multiple-testing experiment displays every p-value in the first family, not only the smallest one. A selector applies either the unadjusted threshold or a Bonferroni threshold. With twenty tests and family alpha 0.05, the adjusted cutoff is 0.05 / 20 = 0.0025. Controls also change the family size, alpha, and number of repeated families. The long-run metric can then be compared with the independent-theory benchmark.

The display must not label a threshold crossing as a scientific discovery. It is a simulated rejection under a known null and exists to show the selection mechanism.

Derivation and reasoning

For an observed statistic t_obs and null model H0, a two-sided p-value can be represented as:

p = P(|T| >= |t_obs| | H0 and the test model)

The probability is about possible test statistics under the model, not about the truth of H0 after seeing data. Reversing that condition would require additional modeling, such as explicit prior probabilities and likelihoods.

For one valid test at level alpha = 0.05, the probability of a Type I error is 0.05 when its null is true. If twenty null tests are independent, the probability that none produces a false positive is:

(1 - alpha)^20 = 0.95^20

Therefore the probability of at least one is:

1 - 0.95^20, approximately 0.6415, or 64.2%.

This is the family-wise error rate for the simple independent setting. Bonferroni controls the family-wise rate by testing each of m hypotheses at alpha / m. The union bound makes this control valid without requiring independence, although it can be conservative when tests are correlated.

False discovery rate answers a different question: the expected proportion of false rejections among rejections under a specified procedure. It should not be presented as interchangeable with family-wise error control.

Assumptions

The 64.2% calculation assumes twenty independent tests whose null hypotheses are all true and whose individual Type I error rates are exactly 5%. Correlated outcomes change the exact family-wise probability. Dependence does not make the search context irrelevant; it means the correction should reflect the analysis structure.

Each underlying test has its own assumptions about sampling, independence, distribution, and test-statistic construction. Multiplicity adjustment cannot rescue an invalid test. Nor can a valid p-value repair biased selection or poor measurement.

The family of tests must be defined by the scientific or decision process. Secretly narrowing the family after inspecting results defeats the purpose. Confirmatory and exploratory analyses can coexist, but they should be labelled honestly.

Failure cases

Do not use 0.049 and 0.051 as opposite scientific conclusions. They are nearly identical summaries under a continuous measure, and neither reports effect size or practical importance. Show estimates and intervals, discuss model fit, and connect magnitudes to a predeclared decision context.

Do not report only the most favorable outcome, subgroup, transformation, stopping time, or model. Selective reporting prevents readers from understanding how many opportunities produced the final number. A correction applied only to the visible tests may still understate the search.

Do not assume Bonferroni is always the required solution. Its strong family-wise protection can be appropriate for a small confirmatory family, while other procedures may fit ordered hypotheses or large exploratory sets. The choice should follow the cost of false and missed findings, not a desire for more or fewer stars.

Do not interpret statistical significance as causality. A randomized design may justify a causal contrast under additional assumptions; a small p-value from observational association does not create random assignment.

Source notes

The American Statistical Association states that p-values do not measure hypothesis truth, effect size, or importance and emphasizes transparency and context. The UC Berkeley statistics text derives how multiplicity increases false positives. This guide combines those principles in an original twenty-outcome synthetic simulation and uses no source data or extended quotation.

Sources checked

These sources support the statistical definitions and boundaries. The scenario, prose, calculations, and figure on this page are original to Count & Chance.

• ASA Statement on Statistical Significance and P-Values, American Statistical Association. Checked 2026-08-19.
• Hypothesis Testing: Multiplicity and Data Mining, University of California, Berkeley Department of Statistics. Checked 2026-08-19.