The hook
Two real treatments for kidney stones. Treatment A cured a higher percentage of patients with small stones. It also cured a higher percentage with large stones. So A is the better treatment — obviously. Right?
It wins in both groups. There are only two groups. It feels airtight. But hold on — let’s actually look at the numbers, and then lump all the patients together.
See it flip
Start on Split by stone size: Treatment A wins both. Then hit Combined and watch what happens to the overall winner.
Small stones → A wins
Large stones → A wins
Treatment A cures a higher share in both groups. Now flip to Combined…
Treatment A beats B in every subgroup, yet loses when you combine them. That’s not a typo or a trick of rounding — it’s a genuine reversal, and it has a name: Simpson’s Paradox.
Make a guess
Before the explanation: how can a treatment win every group but lose the total? What do you think is hiding in these numbers?
The reveal
Large stones are simply harder to cure than small ones — everyone does worse on them. Now look at how the cases were handed out:
Treatment A was handed mostly large (hard) stones; Treatment B mostly small (easy) ones. That imbalance is the whole trick.
Doctors gave the tough large-stone cases mostly to Treatment A, and the easy small-stone cases mostly to B. So A’s overall score is dragged down by all the hard cases it took on, while B’s is propped up by easy ones. The combined average isn’t measuring the treatment — it’s measuring who walked through each door.
The math, gently
Here’s the quiet move that trips everyone up. The combined cure rate is not the average of the two subgroup rates. You don’t add the percentages and halve them. You pool the raw counts:
In effect each group gets a weight equal to how many patients it had — a big group pulls the overall number toward its own rate, a tiny group barely tugs at all.
That weighting is the whole trick. A treatment can win every single subgroup and still lose overall — as long as its wins pile up in the small groups and its losses sit in the big ones. If a treatment was tested mostly on the harder cases (where everybody’s base success is low), those heavy, low-scoring weights drag its combined number down, even while it beats the rival case-for-case.
For the flip to actually happen you need a lurking variable — here, stone size, i.e. case difficulty — that is both linked to the outcome (large stones are harder to cure) and unevenly spread across the treatments (A took most of the large stones). Knock out either condition and the paradox can’t form: a confounder that doesn’t move the outcome, or one that’s split evenly, is harmless.
The fix is the same fix every time: don’t trust the lone pooled average. Split the data by the lurking variable and compare like with like — or, if you must report one number, weight the groups fairly instead of letting the case mix do it for you.
Check yourself
Why can the single pooled cure rate lie about which treatment is better?
What does it mean that the combined cure rate is a 'weighted average'?
What does the paradox actually require, and what's the right fix?
Where it shows up
This isn’t a rare curiosity — it ambushes real decisions all the time. It famously made a university look like it discriminated against women in admissions (it didn’t — women applied to tougher departments). It can make a hospital, a drug, or a sales campaign look better or worse than it is. The lesson is permanent: whenever you’re shown a single average, ask “averaged over what?” — because the groups underneath might be telling the opposite story.