Twenty Tests, One Discovery

The first false hit feels special

A result with p < 0.05 is often translated into one of three incorrect statements: there is less than a 5% chance the null hypothesis is true, there is less than a 5% chance the result happened randomly, or the effect has a 95% chance of being real. A p-value does not provide any of those probabilities.

It measures how incompatible the observed test statistic is with a specified statistical model, usually including a null hypothesis and assumptions about data generation. It is a conditional probability of data at least as extreme as what was observed, calculated as if that null model were true. The American Statistical Association statement also warns that a p-value does not measure effect size or the importance of a result.

Even that interpretation is incomplete when many analyses were attempted. A threshold calibrated for one planned test does not automatically control the chance of a false claim across twenty opportunities.

Watch a whole family, not the winner

Generate twenty independent valid p-values under true null hypotheses. Under a continuous valid null model, each p-value has a uniform distribution between zero and one. This direct generator removes subject-matter distractions and isolates the selection problem. Apply alpha = 0.05 to every outcome and highlight p-values below 0.05.

There are no real participants, products, interventions, or outcomes. The generator sets every null effect to exactly zero. Nevertheless, some batches of twenty show one or more highlighted results. Change the seed and the location of the smallest p-value changes. Repeat the entire twenty-test batch thousands of times and count how often at least one false positive appears.

The linked multiple-testing experiment displays every p-value in the first family, not only the smallest one. A selector applies either the unadjusted threshold or a Bonferroni threshold. With twenty tests and family alpha 0.05, the adjusted cutoff is 0.05 / 20 = 0.0025. Controls also change the family size, alpha, and number of repeated families. The long-run metric can then be compared with the independent-theory benchmark.

The display must not label a threshold crossing as a scientific discovery. It is a simulated rejection under a known null and exists to show the selection mechanism.

A fixed twenty-value rehearsal

Use this authored sequence as a selection audit. It is a teaching fixture, not the output of a study and not evidence that a random generator produced this particular order:

0.184, 0.611, 0.043, 0.728, 0.291, 0.014, 0.907, 0.352, 0.566, 0.082, 0.447, 0.238, 0.775, 0.119, 0.684, 0.331, 0.958, 0.205, 0.493, 0.067

At the unadjusted 0.05 cutoff, tests 3 and 6 cross the line. If only test 6 were shown, its p = 0.014 could look like a lone, planned finding. The complete row shows twenty opportunities. With a Bonferroni family-wise cutoff of 0.05 / 20 = 0.0025, none crosses. The exercise does not prove either decision rule is right for a real project; it makes the consequences of the declared family visible.

Now write down the family before changing the values. Would tests 1 through 20 answer one decision, several ordered decisions, or unrelated exploratory questions? That substantive choice comes before correction. A denominator invented after inspecting the smallest value is not a defensible multiplicity plan.

Why the chance reaches 64.2%

For an observed statistic t_obs and null model H0, a two-sided p-value can be represented as:

p = P(|T| >= |t_obs| | H0 and the test model)

The probability is about possible test statistics under the model, not about the truth of H0 after seeing data. Reversing that condition would require additional modeling, such as explicit prior probabilities and likelihoods.

For one valid test at level alpha = 0.05, the probability of a Type I error is 0.05 when its null is true. If twenty null tests are independent, the probability that none produces a false positive is:

(1 - alpha)^20 = 0.95^20

Therefore the probability of at least one is:

1 - 0.95^20, approximately 0.6415, or 64.2%.

This is the family-wise error rate for the simple independent setting. Bonferroni controls the family-wise rate by testing each of m hypotheses at alpha / m. The union bound makes this control valid without requiring independence, although it can be conservative when tests are correlated.

False discovery rate answers a different question: the expected proportion of false rejections among rejections under a specified procedure. It should not be presented as interchangeable with family-wise error control. The UC Berkeley testing chapter works through how repeated opportunities and data mining change the error question; the sequence above is an independent example created for this guide.

Fill in the analysis card first

Before opening a result table, a confirmatory analysis record should state the outcome definitions, the tests that belong to the family, the direction of each contrast, the stopping rule, the error rate to control, and the treatment of missing data. It should also identify effect sizes and uncertainty summaries that will appear regardless of threshold crossings.

This does not ban exploratory work. It separates exploration from confirmation so a later reader can see which patterns were proposed in advance and which were noticed in the data. A short honest record is more useful than a detailed plan written after the results are known.

Conditions behind the benchmark

The 64.2% calculation assumes twenty independent tests whose null hypotheses are all true and whose individual Type I error rates are exactly 5%. Correlated outcomes change the exact family-wise probability. Dependence does not make the search context irrelevant; it means the correction should reflect the analysis structure.

Each underlying test has its own assumptions about sampling, independence, distribution, and test-statistic construction. Multiplicity adjustment cannot rescue an invalid test. Nor can a valid p-value repair biased selection or poor measurement.

The family of tests must be defined by the scientific or decision process. Secretly narrowing the family after inspecting results defeats the purpose. Confirmatory and exploratory analyses can coexist, but they should be labelled honestly.

A reporting audit for the final table

Do not use 0.049 and 0.051 as opposite scientific conclusions. They are nearly identical summaries under a continuous measure, and neither reports effect size or practical importance. Show estimates and intervals, discuss model fit, and connect magnitudes to a predeclared decision context.

Do not report only the most favorable outcome, subgroup, transformation, stopping time, or model. Selective reporting prevents readers from understanding how many opportunities produced the final number. A correction applied only to the visible tests may still understate the search.

Do not assume Bonferroni is always the required solution. Its strong family-wise protection can be appropriate for a small confirmatory family, while other procedures may fit ordered hypotheses or large exploratory sets. The choice should follow the cost of false and missed findings, not a desire for more or fewer stars.

Do not interpret statistical significance as causality. A randomized design may justify a causal contrast under additional assumptions; a small p-value from observational association does not create random assignment.

Before release, count the outcomes, subgroups, transformations, interim looks, and model variants that were available to the analyst. Compare that inventory with what appears in the report. If only one attractive row remains, explain the selection or restore the omitted context. Then read the effect estimate and interval without looking at the p-value. If the practical conclusion changes solely because a third decimal moved across 0.05, the report is asking the threshold to do too much work.

Limits of this teaching family

The American Statistical Association states that p-values do not measure hypothesis truth, effect size, or importance and emphasizes transparency and context. The UC Berkeley statistics text derives how multiplicity increases false positives. This guide combines those principles in an original twenty-outcome synthetic simulation and uses no source data or extended quotation.

Revision note, August 26, 2026. Added a fixed twenty-value selection rehearsal, a pre-analysis record, a final-table audit, and paragraph-level ASA and UC Berkeley links. The displayed values are an authored teaching fixture, not study results.