How Does Simpson's Paradox Happen?

How does Simpson’s paradox happen?

Simpson’s paradox happens when a relationship in combined data changes direction, or disappears, after the data are separated into relevant groups. A comparison may look favorable overall but unfavorable within every group, or unfavorable overall but favorable within every group. The arithmetic is not necessarily wrong. The problem is that aggregation can hide how observations are distributed across groups.

The name describes a pattern in data, not a special failure of mathematics. It is best treated as a warning to ask what has been combined. Before interpreting an overall rate, consider whether another variable changes the difficulty, risk, or composition of the observations being compared.

The aggregation step can change the question

Suppose a report compares the success rates of two treatments. The overall rate answers a question such as, “What fraction of all treated cases succeeded?” That can be useful for describing the results actually observed. It does not automatically answer, “Which treatment works better for cases of the same difficulty?”

A relevant grouping variable can affect both the treatment assignment and the outcome. For example, one treatment may be used more often for difficult cases, while another is used more often for easier cases. When the data are combined, the treatment used with easier cases may receive a compositional advantage. Conversely, a treatment used more often with difficult cases may look weaker overall even if it performs better at each difficulty level.

This is closely related to conditional and joint percentages in contingency tables. An overall percentage uses a total denominator. A within-group percentage uses the total for a particular group. Changing the denominator changes the comparison, even when every underlying count remains fixed.

A worked synthetic example

The following worked example uses entirely synthetic, illustrative numbers. Imagine two treatments, Treatment A and Treatment B, tested on cases classified before treatment as either easier or harder. The classification is included because it may affect the chance of success.

Every count in this table is synthetic. Within the easier synthetic cases, Treatment A succeeds in 90 out of 100 cases, or 90%, while Treatment B succeeds in 19 out of 20 cases, or 95%. Within the harder synthetic cases, Treatment A succeeds in 9 out of 20 cases, or 45%, while Treatment B succeeds in 1 out of 10 cases, or 10%.

Treatment B therefore has the higher synthetic success rate in both groups. But after combining the groups, Treatment A has 99 successes out of 120 cases, or 82.5%, while Treatment B has 20 successes out of 30 cases, or about 66.7%. The overall comparison reverses the within-group comparison.

Why the reversal occurs

The key is the different mix of case types. In this synthetic example, Treatment A is used mostly on easier cases: 100 of its 120 cases are easier. Treatment B is used mostly on harder cases: 10 of its 30 cases are harder, and its easier-case total is much smaller. Because easier cases have higher success rates for both treatments, a treatment used more often on easier cases receives more favorable weighting in the combined result.

The overall rate is a weighted average of the group-specific rates. The weights are the proportions of cases in each group for that treatment. Since Treatment A and Treatment B have different weights, their overall rates are not formed from the same mixture of easier and harder cases. Comparing those overall rates can therefore mix treatment performance with case composition.

This does not prove that the grouping variable is the only explanation, or that it should always be used to adjust a result. It shows that the choice of comparison matters. A fair interpretation may require comparing treatments within comparable groups, using a justified adjustment method, or designing a study so that the groups are balanced.

Simpson’s paradox is not automatically a causal conclusion

A reversal can signal confounding, but the pattern alone does not establish a causal relationship. A variable may be a genuine source of difficulty, a consequence of the treatment decision, a measurement artifact, or a proxy for several unmeasured factors. The meaning depends on how the data were collected and what question the analysis is intended to answer.

Study design also matters. In a randomized experiment, random assignment is intended to balance relevant characteristics between treatment groups on average, although chance imbalance can still occur in a particular sample. In observational data, treatment choice may be related to patient condition, location, time period, or other factors. Adjusting for those variables can improve comparability, but adjustment requires defensible variables and a suitable method.

There is also no universal rule that says the overall result is wrong and the stratified result is right. If the real question concerns the outcomes among all cases treated in practice, the overall rate may be relevant. If the question concerns performance for cases with comparable difficulty, within-group results may be more informative. Analysts should state which population and comparison their percentages describe.

How to investigate a suspected reversal

Start by recreating the raw counts, not just copying percentages from a chart. For each comparison group, record the number of successes, the total number of observations, and any variables that could affect the outcome. Check that the denominators are clearly defined and that missing observations are handled consistently.

Next, calculate both the combined rates and the rates within plausible groups. Look for a variable that changes the baseline outcome rate or has a noticeably different distribution between the comparison groups. Time period, age, severity, location, prior exposure, and selection into a program are common candidates, but the relevant variable must come from the subject matter rather than from a search for any convenient split.

Then ask whether the groups were defined before the outcome occurred and whether the grouping variable is measured reliably. Compare the interpretation under several reasonable specifications, and report the group sizes alongside the percentages. Very small denominators can make a percentage unstable, even when no paradox is present.

Finally, explain the mechanism in plain language. A strong report says which overall association was observed, which within-group associations were observed, how the group composition differed, and which conclusion is supported by the study design. It should distinguish description from causal claims and acknowledge important variables that were not measured.

Practical checklist

• Identify the denominator behind every reported rate.
• Recalculate the comparison within meaningful, pre-specified groups.
• Check whether the groups have different baseline risks or different sizes.
• Separate an observed association from a causal claim.
• Report counts with percentages and explain which question each result answers.