Lab 12 / Uncertainty
Twenty chances for one false alarm
What happens when a 5% threshold is used twenty times?
Generate complete families of valid null p-values and compare an unadjusted threshold with Bonferroni family-wise control. Change one control at a time, then open the values under the chart before interpreting the picture.
View the values behind this chart
Loading the deterministic simulation...
SEEDInterpret the result
The paired explanation covers the mechanism, assumptions, and cases where this simulation should not be generalized.
01 / Question
What this display answers
A valid null p-value crosses 0.05 about 5% of the time for one planned test. Across twenty independent null tests, the chance that at least one crosses is about 64%.
Bonferroni divides the family threshold by the number of tests. That controls the chance of any false rejection in this simple family, but it does not measure effect size, importance, or the probability that a hypothesis is true.
02 / Reading
What to notice
- Inspect all p-values from the first family, including those that do not cross the line.
- Increase the number of tests. The unadjusted family-wise false-alarm rate rises.
- Apply Bonferroni. The threshold drops and far fewer simulated families contain any rejection.
03 / Mechanism
The model behind it
Every programmed null is true. Under a valid continuous null model, each p-value is uniform between zero and one, so the simulation can represent valid null tests without inventing subject matter.
Tests are independent. The exact unadjusted benchmark is one minus the probability that every p-value stays above the threshold.
04 / Limits
Where the result stops
Real test statistics can be dependent, and a correction cannot rescue biased data, invalid assumptions, or selective reporting outside the declared family.
Threshold crossings in this lab are simulated false rejections, not discoveries or evidence about any real intervention.