Count & ChanceStatistics you can rerun.

Applied casebook

Three case files for checking statistical claims

Each file begins with a compact dataset, asks you to make a decision, and then reveals the calculation that changes the interpretation.

Original teaching data

All three datasets are synthetic and created for this publication. They describe no real organization, product, patient, or population.

3worked case files
18source rows
3downloadable CSVs
CASE 01

Detection under a rare base rate

Quality alert audit

Download CSV

The decision

A synthetic inspection system reviews 10,000 items across two shifts. Management asks whether an alert is reliable enough to trigger an automatic production stop.

27.4%of all alerts are true in this constructed example

The detector catches 90 of 100 defects, but it also raises 238 false alerts among 9,900 non-defects. Sensitivity is high; alert precision is not.

ShiftTrue alertsMissed defectsFalse alertsCorrect clears
A428904,860
B4821484,802
Total90102389,662

Audit tasks

  1. Compute sensitivity for each shift.
  2. Compute the false-positive rate for each shift.
  3. Explain why a 90% sensitivity does not imply 90% alert precision.
Check the calculation

Shift A sensitivity is 42 / 50 = 84%; Shift B is 48 / 50 = 96%. Their false-positive rates are 90 / 4,950 = 1.82% and 148 / 4,950 = 2.99%. Overall precision is 90 / (90 + 238) = 27.4%. The rare 1% defect base rate supplies many more opportunities for a false alert than for a true alert.

CASE 02

An aggregate that reverses the groups

Segmented rollout reversal

Download CSV

The decision

Two synthetic onboarding options are compared for new and returning visitors. Option B has the higher completion rate in both groups, yet Option A wins in the raw total.

84% vs 71%raw totals favor A because the group mix is different

A receives mostly returning visitors, while B receives mostly new visitors. A common 50/50 weighting gives A 75% and B 80%, reversing the aggregate ranking.

OptionVisitor groupCompletedTotalRate
ANew12020060%
AReturning72080090%
BNew52080065%
BReturning19020095%

Audit tasks

  1. Verify both within-group comparisons.
  2. Calculate each option's unadjusted total rate.
  3. Reweight each option to a 50/50 new-returning mix.
Check the calculation

Within groups, B leads 65% to 60% for new visitors and 95% to 90% for returning visitors. Raw totals are 840 / 1,000 = 84% for A and 710 / 1,000 = 71% for B. With equal group weights, A is (60% + 90%) / 2 = 75% and B is (65% + 95%) / 2 = 80%.

CASE 03

Response volume versus representation

Survey response mix

Download CSV

The decision

A synthetic population is evenly divided between Group A, with 80% support, and Group B, with 40% support. A probability sample preserves the mix; a voluntary response sample does not.

60% vs 76%population support versus the voluntary estimate

The voluntary sample is 90% Group A and only 10% Group B. More voluntary responses would narrow random error around 76% without moving the estimate toward the 60% target.

SourceGroupShare of sampleSupport in groupContribution
PopulationA50%80%40 points
PopulationB50%40%20 points
ProbabilityA50%80%40 points
ProbabilityB50%40%20 points
VoluntaryA90%80%72 points
VoluntaryB10%40%4 points

Audit tasks

  1. Reconstruct the population support rate.
  2. Reconstruct the voluntary estimate.
  3. Name the change required to fix the target, not just the precision.
Check the calculation

The population rate is 0.50 x 0.80 + 0.50 x 0.40 = 0.60. The voluntary estimate is 0.90 x 0.80 + 0.10 x 0.40 = 0.76. Increasing the same voluntary sample only makes 76% more precise. The recruitment or weighting design must recover the target population composition.

Reusable worksheet

A six-line decision record

Use this compact record when a statistical claim moves from analysis into a memo, dashboard, or product decision.

  1. 01Claim

    Write the exact decision the number is being used to support.

  2. 02Target

    Name the population, outcome, denominator, and time window.

  3. 03Design

    Record who or what could enter the data and how selection occurred.

  4. 04Calculation

    Keep the formula, subgroup table, and unrounded inputs.

  5. 05Stress test

    Change one consequential assumption and report the direction of movement.

  6. 06Boundary

    State what this analysis cannot establish and what evidence would be needed next.

Choose a learning route