THE ARTICLE · 8 MIN
A misleading number is rarely a false one. Usually every figure is accurate and the story built on them is not — because the comparison is missing, the base is tiny, the sample is small, or the data were filtered before anyone looked.
These ten traps cover most of it. Every worked example below uses the published figures, except one marked as invented, and we have redone the arithmetic so you can follow it.
1. Ignoring the base rate
Here is a classic problem from the psychologists Amos Tversky and Daniel Kahneman, who trace it to their own work in 1972.
A city has two cab companies. 85% of the cabs are Green and 15% are Blue. A cab is involved in a hit-and-run at night. A witness says it was Blue, and tests show the witness identifies colours correctly 80% of the time. What is the probability the cab really was Blue?
The typical answer is about 80%. Tversky and Kahneman report that “The median and modal answer is typically .80”. The correct answer is about 41%.
The arithmetic. Out of every 100 cabs, 15 are Blue and the witness correctly calls 12 of them Blue (15 × 0.8). But 85 are Green and the witness wrongly calls 17 of them Blue (85 × 0.2). So of the 29 cabs the witness would call Blue, only 12 really are. Their own working: “P(B/W) = 12/(12 + 17) = .41.”
The lesson. Evidence has to be weighed against how common something was to begin with. A fairly reliable witness is not enough when, in their words, “the base rate is more extreme than the witness is credible.”
2. Correlation that means nothing
Two things rising and falling together does not show that one causes the other.
Tyler Vigen built a whole project on this, called Spurious Correlations. His site pairs unrelated data series that happen to match — for example, the distance between Uranus and the Moon and electricity generation in Japan, which from 1980 to 2021 correlate at about 0.985. The match is real in the numbers and meaningless in the world. Vigen is open about how he finds them. “I have 25,237 variables in my database,” his site explains. “I compare all these variables against each other to find ones that randomly match up.”
Sometimes the link is real but runs through something else. The textbook Forecasting: Principles and Practice gives the standard illustration: ice-cream sales and drownings rise together, but “They are both caused by a third variable (temperature).”
The check. Ask what else could drive both numbers, and whether the pair was found by searching through many possibilities.
3. Relative change without the base
The EU statistics office, Eurostat, uses a simple example: employment in a country rises from 4.8 million to 5.2 million. That is an increase of 0.4 million people — or, in relative terms, about 8.3% (5.2 ÷ 4.8 = 1.083).
Both descriptions are true. Which one sounds impressive depends on the base. A rise from 2 to 4 is “doubled” and “+100%”, and it is still only 2. When you see a percentage change, find the starting number.
4. Percent versus percentage points
These two are routinely confused, and the difference can be large.
Eurostat’s example: in the EU in 2022, 85.9% of women and 81.4% of men aged 20 to 24 had completed at least upper-secondary education. In its words, “85.9 % minus 81.4 % = 4.5 percentage points.”
But relative to the men’s figure, the women’s rate is about 5.5% higher (85.9 ÷ 81.4 = 1.055). “4.5 points” and “5.5%” describe the same gap. A percentage point is the difference between two percentages; a percent change is a ratio.
5. Survivorship bias
You only see what made it through the filter.
The famous story is Abraham Wald, a statistician at Columbia University’s Statistical Research Group during the Second World War. His work was a series of memoranda written in 1943. When the Center for Naval Analyses reprinted them in 1980, it noted that “this work was never published externally”.
The logic is sound: holes on the planes that came back show damage a plane could survive. Hits in places that brought planes down are under-represented, because those planes never returned to be counted.
Two details are usually lost in the retelling:
- Wald’s bombing-mission example was hypothetical. It begins “Of 400 planes on a bombing mission, 359 return” and states that, “for the observed data of this hypothetical example, the engine area is the most vulnerable”. It was a method for estimating vulnerability, not a report of real bombers.
- The famous picture is modern. Cameron Moll wrote in 2022 that “sometime around 2005 I hastily plotted fictitious red dots on a poorly-chosen commercial aircraft outline”, and that the image widely shared online “is a recreation of my diagram”.
Our survivorship bias guide covers how the same trap shows up in success stories.
6. Simpson’s paradox — when the total says the opposite
In autumn 1973, the University of California, Berkeley admitted a higher share of the men who applied to graduate school than of the women: about 44% of the men and 35% of the women.
It looked like discrimination. When P. J. Bickel, E. A. Hammel and J. W. O’Connell examined it department by department, in a paper published in Science in 1975, the pattern changed. The aggregate data showed “a clear but misleading pattern of bias against female applicants.” Few individual departments showed a significant difference, and once the data were pooled in a way that took each department into account, they found “a small but statistically significant bias in favor of women.”
Why the total flipped. In the six largest departments, most women applied to the four that admitted the smallest share of applicants, while about half the men applied to the two that admitted the largest share. Department A, for example, admitted 62% of male applicants and 82% of female ones — but far more men than women applied there.
The authors did not conclude that there was no problem. They put the gap down not to “any pattern of discrimination on the part of admissions committees, which seem quite fair on the whole”, but to “prior screening at earlier levels of the educational system” — in their account, women were steered towards fields of study that were more crowded.
The check. When a total looks damning or glowing, ask whether the groups being combined are comparable.
7. Regression to the mean
Extreme results tend to be followed by less extreme ones. The standard name for it is regression to the mean.
Daniel Kahneman described in his Nobel biography how he learned this while teaching flight instructors in the Air Force. A seasoned instructor told him that when he praised cadets for a clean aerobatic manoeuvre they generally did worse next time, and when he screamed at them for a bad one they generally did better. Kahneman’s explanation was regression to the mean: praise tends to follow an unusually good attempt and scolding an unusually bad one, and the next attempt tends to be more ordinary either way. His summary: “we are statistically punished for rewarding others and rewarded for punishing them.”
The check. When something improves after an unusually bad result — or worsens after an unusually good one — ask whether it would have moved anyway.
8. Small samples swing wildly
Small groups produce more extreme averages, in both directions.
Howard Wainer showed this in American Scientist with school test scores. Looking at the fifth-grade reading scores of 1,662 Pennsylvania schools, he found that “of the top-scoring 50 schools (the top 3 percent) six were among the smallest 3 percent of the schools.” But the bottom of the list told the same story: “Nine of these (18 percent) were among the 50 smallest schools.”
Small schools were over-represented at both ends. In Wainer’s words, “smaller schools are expected to have higher variance and hence should be over-represented at both extremes.” Tversky and Kahneman had described a related error in 1971: “People have erroneous intuitions about the laws of chance.”
The check. Before trusting a “best” or “worst” ranking, look at how big the groups at the top and bottom are.
9. Bar charts that don’t start at zero
A bar’s length is read as its size. Cut off the bottom of the axis and small differences look large.
The UK Government Analysis Function is direct about it: “We advise that bars on bar charts should always start at zero.” Its reason: “When bars do not start at zero, the distance between bar height or length is exaggerated.”
An invented example. Two values of 50 and 55 differ by 10%. Start the axis at 45, and the bars show 5 against 10 — one looks twice the other.
10. “Average” can hide almost everything
The mean is pulled by extreme values; the median is not.
The Australian Bureau of Statistics uses the retirement ages of 11 people, from 54 to 60. The mean is about 56.6 and the median 57. Change one of the two 60s to 81, and the mean becomes “(54+54+54+55+56+57+57+58+58+60+81 = 644), divided by 11 = 58.5 years” — while the median stays at 57. As the bureau puts it, “The mean is more sensitive to the existence of outliers than the median or mode.”
The check. When you see “the average”, ask whether it is the mean or the median, and how spread out the values are.
The one-minute number check
- Compared with what? A number with no comparison says very little.
- What was the base? Big percentages often sit on small starting numbers.
- Points or percent? They are different figures.
- How big was the sample? Extreme results cluster in small groups.
- What got filtered out before you saw it?
- Does the total hide groups pulling in different directions?
- Mean or median — and where does the axis start?
Sources
- Amos Tversky and Daniel Kahneman, Evidential Impact of Base Rates, Office of Naval Research Technical Report No. 4, 15 May 1981.
- Tyler Vigen, Spurious Correlations (website).
- Rob J. Hyndman and George Athanasopoulos, Forecasting: Principles and Practice, 3rd edition, section 7.8.
- Eurostat, “Statistical concept — Percentage change and percentage points”, Statistics Explained.
- Abraham Wald, Statistical Research Group memoranda on aircraft vulnerability (1943), reprinted by the Center for Naval Analyses (CRC 432, July 1980).
- Cameron Moll, “Abraham Wald and the airplane diagram with red bullet holes – here’s the origin story”, personal website, 24 March 2022.
- P. J. Bickel, E. A. Hammel and J. W. O’Connell, “Sex Bias in Graduate Admissions: Data from Berkeley”, Science 187 (1975).
- Daniel Kahneman, Biographical, The Nobel Prize (2002).
- Howard Wainer, “The Most Dangerous Equation”, American Scientist 95 (2007).
- Amos Tversky and Daniel Kahneman, “Belief in the Law of Small Numbers”, Psychological Bulletin 76 (1971).
- UK Government Analysis Function, “Introduction to data visualisation”, module 5: bar charts.
- Australian Bureau of Statistics, “Measures of central tendency”.
Checked September 2026.
Related: How to check something online: the SIFT method · Logical fallacies cheat sheet · Overcoming survivorship bias
- critical thinking
- statistics
- numeracy
- data literacy