The misconception
After computing a 95% confidence interval from one sample, it is tempting to say, “There is a 95% probability that the true mean lies in this interval.” That statement mixes two different frameworks. In the usual frequentist construction, the population mean is a fixed number. The endpoints are random because a different sample would produce different endpoints.
Once the sample has been drawn and the interval calculated, that particular interval either covers the fixed mean or it does not. The 95% describes the long-run performance of the procedure: under its assumptions, about 95% of intervals produced over repeated sampling will cover the parameter.
This is more than a wording preference. It directs attention to the process that generated the data and to the assumptions that make the advertised coverage plausible.
A synthetic experiment
The linked experiment uses an intentionally simple normal population with a fixed mean of 50 and a known standard deviation of 10. The values are synthetic and are not associated with a real component, maker, warranty, or reliability claim. At the default settings, it draws 200 independent samples of size 25. For each sample, it calculates the mean and a 95% z interval.
The chart draws the first 34 intervals as horizontal lines around their sample means. Blue intervals cross the fixed mean of 50, while coral intervals miss it; a vertical reference line marks the true mean. The metrics and value table summarize the complete batch, including intervals not drawn in the chart. Select New draw and the seed changes, so the exact misses change even though the model and settings do not.
Move the intervals control from 50 toward its maximum of 1,000. The observed coverage proportion should become less erratic and remain near the selected target when the assumptions and implementation are correct. Then change the confidence level among 90%, 95%, and 99%. Higher confidence produces wider intervals because the procedure must capture the fixed target more often in repeated use.
Finally, move the sample-size slider. Larger samples narrow the intervals without changing the nominal confidence level. This demonstrates two separate properties: confidence level controls a procedure’s target coverage, while sample size and variability strongly affect precision.
Derivation and reasoning
For independent observations from a normal population with known standard deviation, the two-sided interval used in the experiment is:
X_bar +/- z_(1 - alpha/2) * sigma / sqrt(n)
Here X_bar is the sample mean, sigma is the known population standard deviation, n is the sample size, and the z value is a standard-normal quantile. For a 95% interval, alpha is 0.05 and the critical value is about 1.96. The lab fixes sigma at 10 so the only Monte Carlo variation comes from the sample means.
The construction starts from a statistic whose sampling distribution is known under the model:
(X_bar - mu) / (sigma / sqrt(n))
The central 95% of its standard-normal distribution lies between the negative and positive critical values. Algebra moves the unknown mu into the interval expression. Before sampling, the random interval has a 0.95 probability of covering mu. Repeating the whole procedure makes that coverage observable.
Width depends on three pieces. The critical value increases when higher confidence is requested. The known sigma sets the population variability. The sqrt(n) term narrows the interval as information grows. Quadrupling sample size halves the standard-error contribution, all else equal.
Assumptions
The exact z result shown here assumes independent observations from a normal population and treats the population standard deviation as known. Known variability is unusual in ordinary analysis; when it is estimated from the same normal sample, a Student’s t interval is commonly used instead. Severe skewness, heavy tails, influential observations, or an estimated standard error require a procedure suited to those conditions. Looking only at the interval does not diagnose those problems.
Random selection or a justified data-generating model connects the sample to the target population. A perfectly calculated interval from a convenience sample does not automatically cover a broader population mean at its nominal rate. Dependence also matters. Treating clustered or repeated measurements as independent makes the standard error too small.
The target must be defined before interpretation. A mean lifetime under one controlled synthetic condition is not necessarily the mean under another condition, and a population mean is not a guarantee for any individual unit.
Failure cases
Do not treat overlap between two separate 95% confidence intervals as a universal significance test. The relationship between interval overlap and a comparison depends on whether estimates are independent, paired, and how the comparison’s standard error is constructed. Calculate an interval for the difference when the difference is the target.
Do not read “not statistically distinguishable from zero” as “proved equal to zero.” A wide interval may include effects of practical importance in both directions. Conversely, a very narrow interval can exclude zero while describing a trivial effect. Show the estimate, interval, units, and meaningful reference values together.
Do not quietly choose the confidence level after seeing which version supports a preferred conclusion. The level and analysis plan should be selected for the purpose before inspecting outcomes. Repeatedly trying subgroups, endpoints, or models introduces selection that the single reported interval does not capture.
Source notes
The NIST handbook explicitly describes confidence as a property of repeated interval construction and supplies mean-interval formulas for known and unknown variability. This guide paraphrases that interpretation and uses the site’s original known-sigma simulation to isolate coverage. It does not reproduce a NIST case study or infer anything about real manufactured components.
Keep these distinctions
- The confidence level belongs to the interval-producing method under stated assumptions.
- A correctly computed 95% interval can miss the fixed parameter.
- Coverage, interval width, and study design must be considered together.
Sources checked
These sources support the statistical definitions and boundaries. The scenario, prose, calculations, and figure on this page are original to Count & Chance.
- 1.3.5.2. Confidence Limits for the Mean, National Institute of Standards and Technology. Checked 2026-08-19.
- 7.1.4. What Are Confidence Intervals?, National Institute of Standards and Technology. Checked 2026-08-19.