When Every Group Improves but the Total Gets Worse

The reversal in four cells

If version B has a higher completion rate than version A among new users and also among returning users, it seems impossible for B to have a lower completion rate overall. The apparent contradiction disappears when the two versions contain different proportions of new and returning users.

An overall rate is not an unweighted summary of subgroup rates. Each subgroup contributes in proportion to its size. A version used mostly by a high-completion group can lead overall even when it trails inside every comparable group. This pattern is known as Simpson’s paradox, although the probability arithmetic itself is entirely consistent. The Stanford Encyclopedia treatment distinguishes the numerical reversal from the harder question of which comparison has the right causal interpretation.

Audit the fictional interface test

The linked experiment compares fictional options A and B across two groups labelled novice and returning. A always has 1,000 expected cases, including 20% novice cases. Its programmed completion rates are 60% for novice cases and 90% for returning cases, so A’s overall rate is 84%.

At the default settings, B’s within-group edge is five percentage points. B therefore completes at 65% among novice cases and 95% among returning cases. However, B’s mix is 80% novice and 20% returning, producing an overall rate of 71%. B leads inside both groups but trails A overall because the lower-rate novice group receives much more weight under B.

Every user and outcome is synthetic. The extreme imbalance is intentional so the weighting mechanism is easy to inspect. It is not evidence about actual interfaces, customer behavior, onboarding, or product experiments.

The experiment provides two controls. B novice share moves B’s novice proportion from 10% to 95%, while A remains fixed at 20%. B within-group edge adds the selected one-to-ten-point advantage over A in both groups; at the highest setting, the returning rate is capped at 99.9% rather than allowed to exceed 100%. Six bars show A and B inside each group and overall; the metrics report both overall rates and whether a reversal is present. Open the value table to verify the group rates rather than relying only on bar length.

First leave the edge at five points and lower B’s novice share until the reversal disappears. Then return to a high novice share and increase the edge. A sufficiently large within-group advantage can overcome more of the composition difference. These operations distinguish two quantities that an aggregate rate combines: performance within the defined groups and the proportions assigned to those groups.

The default counts fit on one review sheet:

Check each comparison before looking at the totals. B completes 5 more cases per 100 within both rows. Then add the cells: A has 840 / 1,000 = 84%, while B has 710 / 1,000 = 71%. The arithmetic does not contradict itself. The novice row carries four times as much weight for B as it does for A.

Why the weights flip the result

For version A, let p_Ag be the completion rate in group g and w_Ag be that group’s share of all A observations. The aggregate rate is:

p_A = sum over groups of (w_Ag * p_Ag)

Version B has its own weights:

p_B = sum over groups of (w_Bg * p_Bg)

Even if p_Bg > p_Ag for every group, p_B can be lower than p_A because w_Bg and w_Ag differ. The comparison combines two changes: subgroup performance and subgroup composition.

Using common weights creates a standardized descriptive comparison. For example, at the default five-point edge, give both options 50% novice and 50% returning cases. A’s standardized rate becomes (0.50 * 0.60) + (0.50 * 0.90) = 0.75. B’s becomes (0.50 * 0.65) + (0.50 * 0.95) = 0.80. The common weighting reflects B’s higher rate inside each group. This calculation is explained here for context; the current interactive display reports each option’s own selected composition rather than adding a standardized view.

Standardization does not, by itself, prove that switching an individual from A to B causes improvement. It answers a defined descriptive question: what would the weighted rates be under this chosen composition? Causal interpretation needs assumptions about assignment, confounding, measurement, and interference. The UC Berkeley discussion of experiments and observational studies is useful here because it keeps assignment and confounding in view while working through grouped comparisons.

Reweight the same four cells

Give both versions a common review population of 500 novice and 500 returning cases. Keep every group-specific rate unchanged.

• A would complete 300 + 450 = 750 cases, or 75%.
• B would complete 325 + 475 = 800 cases, or 80%.

Now B’s within-group edge and overall edge point in the same direction because the weights are held constant. This is a transparent descriptive standardization, not a reconstruction of what would have happened to any particular user. A reviewer should record who chose the 50/50 target and why that population answers the decision at hand.

Conditions the table relies on

Subgroups must use the same definition across versions. If “returning” means a different amount of experience under A and B, the stratified comparisons do not compare like with like. Each cell also needs enough information for a stable rate; a perfect score from one observation should not be treated as strong evidence.

The example treats completion as consistently measured. Missing outcomes, duplicate sessions, or different follow-up windows can alter denominators. The aggregate and subgroup tables should therefore display counts as well as percentages.

Most importantly, the grouping variable must be interpreted in the data-generating process. Experience might confound an observational comparison because it affects both version exposure and completion. But conditioning on a variable caused by both exposure and outcome-related factors can introduce collider bias. “Always split the data” is no safer than “never split the data.”

When grouping creates a new error

Do not select the aggregation level that produces the preferred story. Report why the groups matter, show both aggregate and subgroup counts, and define any standard weights before examining which version wins.

Do not infer discrimination, product quality, or treatment efficacy from a reversal alone. Simpson’s paradox alerts the analyst to composition and confounding; it does not identify the correct causal adjustment automatically. Domain knowledge and study design decide which comparison answers the question.

Do not average percentages without their denominators. The simple average of 80% from 100 observations and 10% from 10 observations is 45%, but the combined rate is 81 / 110, about 73.6%. A percentage carries its sample size with it.

Boundary between example and evidence

The UC Berkeley text uses grouped admission data to explain confounding and weighted rates, while the Stanford Encyclopedia describes the broader mathematical and causal forms of Simpson’s paradox. This guide uses a newly constructed two-interface example. It borrows the principle, not the source data, phrasing, or conclusions.

Revision note, August 26, 2026. Added a four-cell completion audit, the common-population reweighting exercise, and paragraph-level links separating the arithmetic reversal from causal interpretation. No observed user data were added.