Cognitive Biases, Checked
14 Famous Findings and Whether They Held Up

Held up. Described by Tversky and Kahneman in 1974. When the Many Labs project tested anchoring questions — where a high or low number is stated in the question itself — across 36 samples, the authors wrote that the original studies had produced underestimates of some effects, anchoring among them. That result is about numbers given in the question; it does not show that any random number you happen to see affects you.

SOURCE Klein et al., Many Labs 1, Social Psychology (2014).

Held up. A 2021 close replication of Fischhoff's 1975 study 'found support for hindsight bias in retrospective judgments'.

SOURCE Chen et al., Journal of Experimental Social Psychology 96 (2021).

Real, but it is a family of effects, not one finding. One well-measured version: a meta-analysis of 51 experiments found people rated identical information more favourably when it supported their political side, a bias its authors called robust and found on both sides. There is no single canonical 'confirmation bias study' to replicate.

SOURCE Ditto et al., Perspectives on Psychological Science (2019).

Held up, at about half the original size. Tversky and Kahneman showed in 1981 that framing the same problem differently produces 'predictable shifts of preference'. In Many Labs 1 the gain-versus-loss framing effect replicated, but the effect came back at roughly half the size originally reported.

SOURCE Tversky & Kahneman, Science (1981); Klein et al. (2014).

Replicated as a small effect in a hypothetical scenario. Many Labs 1 tested a written sunk-cost scenario (based on a 2009 study by Oppenheimer, Meyvis and Davidenko). Across all samples combined it replicated, but only about half of the 36 individual samples found a significant effect. It was a hypothetical choice, not a decision with real money at stake.

SOURCE Klein et al. (2014).

Smaller than claimed. A 2018 study, focusing on children whose mothers had not completed college, found the link with later achievement was 'only half the size of those reported in the original studies and was reduced by two thirds' once family background, early ability and the home environment were taken into account.

SOURCE Watts, Duncan & Quan, Psychological Science (2018).

Real, but usually misread. A meta-analysis confirmed that each individual is less likely to help when others are present, but found the effect weaker in dangerous emergencies. And when researchers studied CCTV footage of real public conflicts, 'in 9 of 10 public conflicts, at least 1 bystander' did something to help.

SOURCE Fischer et al., Psychological Bulletin (2011); Philpot et al., American Psychologist (2020).

The hormone and behaviour claims failed; the feeling replicated in that study. A 2015 replication with 200 people 'found no significant effect of power posing on hormonal levels or in any of the three behavioral tasks', but did find people felt more powerful. The original study's first author, Dana Carney, has since written: 'I do not believe that "power pose" effects are real.'

SOURCE Ranehill et al., Psychological Science (2015); Carney, 'My position on Power Poses'.

The pen study did not replicate; a small effect of facial feedback seems real. A 17-lab replication of the pen study found a rating difference of just 0.03 units. A meta-analysis found facial feedback's overall effect 'significant but small', and a later multi-country study found the evidence 'less conclusive' for the pen-in-mouth method specifically.

SOURCE Wagenmakers et al. (2016); Coles, Larsen & Lench (2019); Coles et al. (2022).

Genuinely contested. A 2018 review argued that current evidence 'does not support that losses, on balance, tend to be any more impactful than gains'. Studies with 17,720 people in total pushed back in 2020, finding people 'of all knowledge and experience levels were loss averse'. A 2024 meta-analysis of 607 estimates put the average loss aversion coefficient at 1.955 — losses weighing roughly twice as much as gains. A second 2024 meta-analysis, limited to studies that fitted prospect theory to choices between risky gambles, found that 'much of the data are of poor quality' and put the figure at 1.31 (95% CI 1.10 to 1.53). We lean towards loss aversion being real on average, because both meta-analyses put it above 1; its size is still argued over, with the two estimates at about 1.3 and about 2.

SOURCE Gal & Rucker (2018); Mrkva et al. (2020); Brown, Imai, Vieider & Camerer, Journal of Economic Literature (2024); Walasek, Mullett & Stewart, Journal of Economic Psychology (2024).

The overestimation is real; its size and cause are disputed. In the 1999 study, the lowest scorers on average put themselves around the 62nd percentile when they were in the 12th. Critics argue much of the famous pattern is a statistical artefact, and one 2020 paper concluded the effect 'may be much smaller than reported previously'. A large 2021 study still found low performers were less able to judge whether their answers were right, in grammar and logical reasoning. Our reading: low performers overestimating themselves is well supported; how much of the classic pattern is real rather than statistical is an open question.

SOURCE Kruger & Dunning (1999); Gignac & Zajenkowski, Intelligence (2020); Jansen, Rafferty & Griffiths, Nature Human Behaviour (2021).

Probably smaller than claimed. A 2015 meta-analysis of studies with schoolgirls found an average effect, but warned that 'publication bias might seriously distort the literature'. A pre-registered study of 2,064 Dutch students aged 13 to 14 then 'found neither an overall effect of stereotype threat on math performance' among girls, nor any moderated effect, and a 2019 meta-analysis of settings closer to real exams, covering women and ethnic minorities, concluded the effect 'may range from negligible to small'. We lean towards a small or negligible effect, because the pre-registered study and the publication-bias warning both point that way. The pre-registered study was about girls and maths; the 2015 meta-analysis covered girls on maths, science and spatial tests; the 2019 meta-analysis covered women and ethnic minorities on cognitive ability tests.

SOURCE Flore & Wicherts, Journal of School Psychology (2015); Flore, Mulder & Wicherts (2018); Shewach, Sackett & Quint, Journal of Applied Psychology (2019).

The strong version failed; a very small effect is disputed. A 23-lab registered replication found an effect whose confidence interval 'encompassed zero'. A 36-site study led by Vohs and colleagues reported that its 'preregistered analyses did not find evidence for a depletion effect', though an exploratory analysis of its full sample found a small significant effect. A preregistered 12-lab study, meanwhile, found 'a small and significant ego depletion effect'. Our reading: the popular idea that willpower drains like a fuel tank is not supported; whether a tiny effect exists is still open.

SOURCE Hagger et al. (2016); Vohs et al., Psychological Science (2021); Dang et al., Social Psychological and Personality Science (2020).

The original finding has not been confirmed. A 2012 replication using automated timing 'failed to show priming'. In a second experiment the slower walking appeared 'only when experimenters believed participants would indeed walk slower' — pointing to the experimenters' expectations rather than the words. We found no large, pre-registered replication either way.

SOURCE Doyen, Klein, Pichon & Cleeremans, PLoS ONE (2012).