The misconception
If version B has a higher completion rate than version A among new users and also among returning users, it seems impossible for B to have a lower completion rate overall. The apparent contradiction disappears when the two versions contain different proportions of new and returning users.
An overall rate is not an unweighted summary of subgroup rates. Each subgroup contributes in proportion to its size. A version used mostly by a high-completion group can lead overall even when it trails inside every comparable group. This pattern is known as Simpson’s paradox, although the probability arithmetic itself is entirely consistent.
A synthetic experiment
The linked experiment compares fictional options A and B across two groups labelled novice and returning. A always has 1,000 expected cases, including 20% novice cases. Its programmed completion rates are 60% for novice cases and 90% for returning cases, so A’s overall rate is 84%.
At the default settings, B’s within-group edge is five percentage points. B therefore completes at 65% among novice cases and 95% among returning cases. However, B’s mix is 80% novice and 20% returning, producing an overall rate of 71%. B leads inside both groups but trails A overall because the lower-rate novice group receives much more weight under B.
Every user and outcome is synthetic. The extreme imbalance is intentional so the weighting mechanism is easy to inspect. It is not evidence about actual interfaces, customer behavior, onboarding, or product experiments.
The experiment provides two controls. B novice share moves B’s novice proportion from 10% to 95%, while A remains fixed at 20%. B within-group edge adds the selected one-to-ten-point advantage over A in both groups; at the highest setting, the returning rate is capped at 99.9% rather than allowed to exceed 100%. Six bars show A and B inside each group and overall; the metrics report both overall rates and whether a reversal is present. Open the value table to verify the group rates rather than relying only on bar length.
First leave the edge at five points and lower B’s novice share until the reversal disappears. Then return to a high novice share and increase the edge. A sufficiently large within-group advantage can overcome more of the composition difference. These operations distinguish two quantities that an aggregate rate combines: performance within the defined groups and the proportions assigned to those groups.
Derivation and reasoning
For version A, let p_Ag be the completion rate in group g and w_Ag be that group’s share of all A observations. The aggregate rate is:
p_A = sum over groups of (w_Ag * p_Ag)
Version B has its own weights:
p_B = sum over groups of (w_Bg * p_Bg)
Even if p_Bg > p_Ag for every group, p_B can be lower than p_A because w_Bg and w_Ag differ. The comparison combines two changes: subgroup performance and subgroup composition.
Using common weights creates a standardized descriptive comparison. For example, at the default five-point edge, give both options 50% novice and 50% returning cases. A’s standardized rate becomes (0.50 * 0.60) + (0.50 * 0.90) = 0.75. B’s becomes (0.50 * 0.65) + (0.50 * 0.95) = 0.80. The common weighting reflects B’s higher rate inside each group. This calculation is explained here for context; the current interactive display reports each option’s own selected composition rather than adding a standardized view.
Standardization does not, by itself, prove that switching an individual from A to B causes improvement. It answers a defined descriptive question: what would the weighted rates be under this chosen composition? Causal interpretation needs assumptions about assignment, confounding, measurement, and interference.
Assumptions
Subgroups must use the same definition across versions. If “returning” means a different amount of experience under A and B, the stratified comparisons do not compare like with like. Each cell also needs enough information for a stable rate; a perfect score from one observation should not be treated as strong evidence.
The example treats completion as consistently measured. Missing outcomes, duplicate sessions, or different follow-up windows can alter denominators. The aggregate and subgroup tables should therefore display counts as well as percentages.
Most importantly, the grouping variable must be interpreted in the data-generating process. Experience might confound an observational comparison because it affects both version exposure and completion. But conditioning on a variable caused by both exposure and outcome-related factors can introduce collider bias. “Always split the data” is no safer than “never split the data.”
Failure cases
Do not select the aggregation level that produces the preferred story. Report why the groups matter, show both aggregate and subgroup counts, and define any standard weights before examining which version wins.
Do not infer discrimination, product quality, or treatment efficacy from a reversal alone. Simpson’s paradox alerts the analyst to composition and confounding; it does not identify the correct causal adjustment automatically. Domain knowledge and study design decide which comparison answers the question.
Do not average percentages without their denominators. The simple average of 80% from 100 observations and 10% from 10 observations is 45%, but the combined rate is 81 / 110, about 73.6%. A percentage carries its sample size with it.
Source notes
The UC Berkeley text uses grouped admission data to explain confounding and weighted rates, while the Stanford Encyclopedia describes the broader mathematical and causal forms of Simpson’s paradox. This guide uses a newly constructed two-interface example. It borrows the principle, not the source data, phrasing, or conclusions.
Keep these distinctions
- An aggregate rate is a weighted average of group-specific rates.
- Different group weights can reverse the direction of an aggregate comparison.
- Whether to condition on a group requires a substantive causal question, not just arithmetic.
Sources checked
These sources support the statistical definitions and boundaries. The scenario, prose, calculations, and figure on this page are original to Count & Chance.
- Experiments and Observational Studies: Simpson's Paradox, University of California, Berkeley Department of Statistics. Checked 2026-08-19.
- Simpson's Paradox, Stanford Encyclopedia of Philosophy, Stanford University. Checked 2026-08-19.