Two fictional print shops publish their same-day completion rates. Shop A finishes routine orders more reliably than Shop B. It also finishes complex orders more reliably. Then someone combines the records, and Shop B appears to win by a huge margin.
No arithmetic has been falsified. No order has changed category. The surprise comes from how many of each kind of order the shops received. This is a way into Simpson’s paradox: an association seen within groups can reverse when those groups are combined.
Open the order books before trusting the headline
Here is the entire invented dataset. “Completed” means finished within the same-day deadline. Routine and complex describe the order before work begins. The numbers are deliberately chosen for a clear calculation; they are not reports about real businesses.
| Order type | Shop A completed/total | Shop B completed/total |
|---|---|---|
| Routine | 18/20 = 90% | 128/160 = 80% |
| Complex | 32/160 = 20% | 2/20 = 10% |
| All orders | 50/180 ≈ 27.8% | 130/180 ≈ 72.2% |
A has the higher rate in each type. B has the higher pooled rate. Notice the denominators: A receives mostly complex orders; B receives mostly routine ones. Since routine orders have much higher completion rates at both shops, B’s order mix gives that easier category far more weight.
The overall rate is not the simple mean of the two percentages. For A, the routine rate applies to twenty orders and the complex rate to 160. Giving each category fifty percent weight would describe a different hypothetical mix, not A’s actual workload.
Change the order mix
Hold category rates fixed and change the routine-order weights; compare the pooled results.
Change the mix while holding the rates still
The model keeps the four within-category rates fixed and changes the proportion of routine orders assigned to each shop. Begin with A at one ninth routine and B at eight ninths routine, matching the table’s proportions. Then give both shops a fifty-fifty mix.
With equal weighting, A’s combined rate becomes 55 percent and B’s becomes 45 percent. The ten-percentage-point advantage in each category is now reflected in the standardized totals. This calculation answers an explicit hypothetical question: what rates would these fixed category figures imply under a shared mix?
It does not retroactively turn A’s fifty actual completed orders into ninety-nine. Standardization is a comparison constructed from rates and selected weights. It is useful precisely because you can state the weights and the question instead of silently changing the denominator.
Two true summaries can answer different questions
If you want to describe the total fraction of orders completed by each shop during this fictional period, the pooled figures are relevant. B completed a larger share of its actual workload. That is a true description of what happened in the dataset.
If you want to compare the shops for an order of a known type, the within-category rates supply different information. A routine order belongs in the routine comparison. Pooling mostly complex work at A with mostly routine work at B can obscure the distinction you need.
If you want to know what would happen if you moved an order from one shop to the other, neither descriptive table alone proves the answer. The orders within each category might differ in other important ways. Perhaps the labels are broad, customers select shops for different reasons, or the deadline is measured inconsistently. Our tidy example establishes the mathematical reversal, not a causal performance estimate.
The important question is how the groups were made
Simpson’s 1951 paper addressed interpretation in contingency tables. Later discussions, including Judea Pearl’s analysis, emphasize that the right comparison cannot always be selected from the numbers alone. You need to know how the variables relate to the process that produced the data.
“Always split the data” is therefore as unreliable as “Always use the biggest total.” A grouping variable might represent a relevant pre-existing difference, an intermediate result, or a selection rule introduced after the events of interest. Those situations require different reasoning.
Hernán, Clayton, and Keiding’s research likewise argues for subject-matter knowledge and a clear causal question. The printed table is an endpoint of choices about collection and grouping. Good analysis has to recover enough of that process to understand what a comparison means.
A practical reading exercise
Take an ordinary claim about a percentage: completion, response, renewal, or acceptance. Write down its numerator and denominator in words. “Orders finished today divided by all orders received today” is already much clearer than “success rate.”
Then identify one plausible factor that changes the baseline rate. Is the work routine or complex? Were responses requested through the same channel? Did the groups face the same deadline? Do not immediately slice the data into every possible category. First decide why a proposed grouping could matter to the question.
Finally ask whether the headline compares like with like, describes actual totals, or implies an intervention. These are distinct tasks. A careful writer can show more than one summary and explain why each is included, rather than choosing the table that produces the preferred winner.
Return to the two print shops. The reversal looks impossible only while the category sizes remain hidden. Once the denominators are visible, the mystery becomes a weighted-average calculation. The deeper lesson survives the arithmetic: a percentage tells you something about a particular collection of cases, and understanding that collection is part of understanding the number.
Follow the evidence: E. H. Simpson: the original 1951 paper; Judea Pearl: Understanding Simpson’s Paradox; Hernán, Clayton and Keiding: The Simpson’s paradox unraveled.
Explore all 80 new articles · Browse the complete illustrated collection
Explore more art, ideas, experiments, and independent projects at Incoprea.
