The Law of Truly Large Numbers, Explained

The law of truly large numbers pdf: the central idea

The phrase “the law of truly large numbers” describes a practical warning about chance. If people look for unusual patterns across a very large number of observations, comparisons, or experiments, some unusual-looking results will appear even when nothing special is happening. The phrase is often associated with Persi Diaconis and Frederick Mosteller, who discussed how large collections of opportunities create surprising coincidences.

This idea is related to, but different from, the law of large numbers. The law of large numbers concerns how averages tend to stabilize as a sample grows under appropriate conditions. The law of truly large numbers concerns the probability of finding at least one rare-looking event when there are many opportunities to find one. A result can be unusual in isolation but unsurprising as part of a very large search.

Why many tests create false alarms

A statistical test usually begins with a null hypothesis: a statement such as “there is no association” or “the treatment has no effect.” A threshold is then chosen for deciding when the observed result would be sufficiently unusual under that null hypothesis. A conventional threshold can be useful, but it applies to each test separately.

If one test has a small probability of producing a false alarm under the null hypothesis, repeating that test many times increases the chance that at least one false alarm will occur. The exact calculation depends on independence and on how the tests are related. For independent tests, the probability of avoiding all false alarms is the product of the individual probabilities of avoiding them. The probability of at least one false alarm is the complement of that product.

The tests do not have to be literally identical. Searching across many outcomes, subgroups, time periods, measurements, model specifications, or survey questions can create the same problem. The search itself increases the number of chances for a result to cross a threshold.

A worked synthetic example

Consider this entirely synthetic illustration. An analyst runs one thousand synthetic statistical tests, each using a five-percent threshold, while assuming that the null hypothesis is true for every test. The numbers are invented for teaching and do not represent a real dataset, medical study, business result, or published finding.

For one synthetic test, the chance of a false alarm is five percent. If the one thousand synthetic tests were independent, the chance that none would produce a false alarm would be calculated as 0.95 raised to the 1,000th power. The chance of at least one false alarm would therefore be one minus that quantity, which is extremely close to one. The expected number of false alarms in this synthetic setup would be 50, calculated as 1,000 multiplied by 0.05.

That does not mean exactly 50 false alarms must occur. It means 50 is the average expected count across many repetitions of the same synthetic process. A particular repetition might produce fewer or more. The important point is that finding a small p-value somewhere in the collection would not, by itself, establish a real effect.

Real tests are often correlated. For example, nearby measurements or overlapping groups may contain much of the same information. Correlation changes the calculation, but it does not remove the basic warning: a broad search can make an apparently exceptional result easier to obtain by chance.

Searching after seeing the data

The law of truly large numbers becomes especially important when analysis decisions are made after looking at the data. An analyst might inspect many outcomes and report only the most interesting one. Or the analyst might try several ways to define a group, exclude unusual observations, select a time window, or specify a statistical model, then emphasize the version that produces the strongest result.

This practice is sometimes called data dredging, p-hacking, or researcher degrees of freedom. The labels refer to different aspects of the same general risk: the reported test may not reflect the full number of opportunities that were examined. A p-value is interpreted relative to a testing procedure, so hiding the search process can make evidence appear stronger than it is.

This issue is not solved by saying that the chosen test was performed only once. If the choice was made from many candidate tests because it looked promising, the earlier candidates still matter for interpreting the result. A transparent report should explain the outcomes and analyses considered, not only the selected result.

How statisticians control multiplicity

One response is to define the primary question and analysis before examining the outcome data. A preregistered plan can identify the main outcome, the comparison of interest, the subgroup definitions, and the rule for handling missing observations. This does not guarantee a correct conclusion, but it limits the number of undisclosed choices.

Another response is to adjust for multiple comparisons. The Bonferroni method, described by NIST, uses a conservative threshold based on the number of tests. Other procedures control different quantities, such as the familywise error rate or the false discovery rate. The appropriate method depends on the purpose of the study and the relationship among the tests.

Adjustment is not a substitute for scientific judgment. A very large study may detect a trivial effect with strong statistical evidence, while a small study may miss an important effect. Researchers should report effect sizes, uncertainty intervals, the number of analyses performed, and whether the result was confirmed in new data.

Reading surprising claims carefully

When a headline presents an unusual association, ask how many opportunities were available for finding it. Were many outcomes measured? Were several populations, time periods, or models explored? Was the result predicted in advance, or did it emerge during an exploratory search? These questions help distinguish a planned test from a result selected after the fact.

A surprising result can still be valuable when it is clearly labeled as exploratory. Exploratory analysis is useful for generating hypotheses. The next step is to test the hypothesis with a new dataset or a new experiment, using a plan that is specified before the confirmatory analysis.

Practical checklist:

• Identify the main hypothesis before focusing on the result.
• Count the outcomes, subgroups, time windows, and model choices that were examined.
• Check whether all tested comparisons were reported or whether only favorable results were shown.
• Look for a multiplicity adjustment when many formal tests were performed.
• Treat an unexpected finding as a hypothesis to confirm, especially when it was discovered through broad searching.