Confirmation Bias
Confirmation bias is the tendency to favour information that supports what you already believe. In CRO it quietly inflates winning variants, distorts review reading, and turns inconclusive tests into "wins".
Confirmation Bias
The tendency to seek, weight, and remember information that confirms what you already believe — and dismiss what contradicts it.
Confirmation bias is a systematic error in how people gather and evaluate evidence. Instead of treating new data on its merits, the brain over-weights signals that agree with the existing belief and under-weights or rationalises away signals that don't.
In an e-commerce context it shows up in two places at once. Shoppers reading product reviews skim for the verdict they're already leaning toward, and CRO teams reading test results unconsciously protect the variant they designed. Both effects are well documented and both cost money — one inflates returns, the other inflates false-positive launches.
On the shopper side, confirmation bias is why a customer who has half-decided to buy a moisturiser will read the five-star reviews carefully and scroll past the two-star ones. They aren't lying to themselves on purpose — the positive reviews simply feel more relevant, more representative, more 'true'. The same store layout produces different conversion rates depending on which belief the visitor walked in with.
On the operator side, the bias attacks experimentation. A team that spent three weeks designing a new PDP layout has a strong prior that the new layout works. When the A/B test comes back at p=0.12, the temptation is to call it a directional win, extend the test until it crosses 0.05, or carve out the segment where it did win. All three moves are forms of confirmation bias eating the discipline of the test. It's one of the most common cognitive biases in CRO programs, and the hardest to spot in yourself.
posterior_odds = prior_odds * likelihood_ratio
prior_odds
Prior odds
Your belief before seeing the data, expressed as odds (a 70% belief = 70/30 = 2.33).
likelihood_ratio
Likelihood ratio
How much more likely the observed data is under the hypothesis vs the null.
posterior_odds
Posterior odds
Your updated belief after seeing the data, in odds form.
An apparel store's CRO lead believes a new sticky add-to-cart bar will lift mobile conversion. She's 80% confident going in. The test runs and produces a likelihood ratio of 1.5 — mildly supportive but far from conclusive. A confirmation-biased reading rounds this to 'it worked'; the Bayesian reading is more honest.
Prior odds (80% belief): 4.0
Likelihood ratio from test: 1.5
→ Posterior odds = 6.0 (≈86% belief)
The data moved her belief from 80% to 86% — a real but small update. Confirmation bias would have read the same result as 'confirmed, ship it'. The math says: ship cautiously, or run more traffic.
The point of the formula isn't to do Bayesian arithmetic on every test. It's to remember that a weak signal plus a strong prior produces a slightly-less-weak signal — not a confirmed hypothesis. Confirmation bias is what happens when you skip the multiplication step and just keep the prior.
Where confirmation bias shows up in a typical CRO program, and how often
| Failure mode | Where it happens | Estimated frequency | Cost |
|---|---|---|---|
| Peeking + early stop | Calling a test the moment p<0.05 appears | 1 in 3 tests | 30-50% false-positive rate |
| Segment hunting | Slicing data until one segment 'wins' | 1 in 4 tests | Inflated launch list, no real lift |
| Selective review reading | Quoting only the reviews that support the redesign | Most qual reviews | Wrong hypothesis prioritised |
| Defending the design | Re-running 'fairer' tests after a loss | 1 in 5 losing tests | 2-4 weeks of test velocity |
| Anchored hypothesis | Ignoring drop-off data that contradicts the brief | Most quarterly roadmaps | Tests target the wrong step |
The fixes are structural, not motivational. Pre-register the metric and the sample size before you look at the data. Have a second person — ideally one who didn't design the variant — sign off on the read. Pull drop-off data before writing hypotheses, not after, so the funnel evidence shapes the brief instead of the brief shaping which evidence you notice.
Confirmation bias FAQ
It's the habit of paying more attention to evidence that supports what you already think, and less to evidence that doesn't. Everyone does it — it's not a character flaw, it's how the brain conserves effort. The cost only shows up when you're making decisions that depend on reading data accurately.
It pushes teams to stop tests early when the favoured variant is ahead, to keep running them when it's behind, and to slice the data until a winning segment appears. The result is a launch pipeline full of changes that looked like wins but won't replicate. Pre-registering your stopping rule is the single most effective fix.
No — confirmation bias is one specific type within the broader family of cognitive biases. Others include anchoring, availability bias, and the sunk-cost fallacy. They overlap in CRO contexts: a team anchored on a redesign will then confirmation-bias its way through the test results.
Shoppers who arrive with positive intent skim for positive reviews and treat negative ones as outliers; sceptical shoppers do the opposite. This is why average star ratings predict conversion less reliably than review diversity and recency. Surfacing balanced reviews — including critical ones the product addresses — often lifts trust more than hiding them.
Indirectly, yes. The bias doesn't change the p-value, but it changes which tests you stop, which segments you report, and which results you remember. Across a year of tests, those choices systematically inflate the apparent win rate well above the true rate.
HARKing — Hypothesising After Results are Known — is one specific behaviour driven by confirmation bias. You see a surprising segment win, then write a 'hypothesis' that predicted it, then present the test as confirming that hypothesis. Confirmation bias is the underlying tendency; HARKing is one of its most damaging expressions in CRO.
Pre-register the primary metric, sample size, and stopping rule before the test launches. Have results reviewed by someone who didn't design the variant. Keep a public log of tests that lost or were inconclusive — teams that only celebrate wins build cultures where losses get rationalised away.
Heavily. Session replays and user interviews are especially vulnerable because the data is rich enough that you can always find a clip supporting your view. The fix is to define what you're looking for before you watch the sessions, and to count occurrences rather than collect anecdotes.
Yes. Small samples produce noisy results, and noise is exactly what confirmation bias preys on — there's always a slice of the data that looks like a win. Running tests to proper sample size is partly a statistical requirement and partly a discipline that removes the room for bias to operate.
When hypotheses are generated from actual drop-off data rather than from team intuition, the starting prior is grounded in funnel evidence instead of in what someone wants to be true. That doesn't eliminate confirmation bias at the readout stage, but it stops the test roadmap itself from being built around it.
Test ideas before you ship them
Run unlimited A/B tests, attach hypotheses to outcomes, and build a searchable archive of what works — and what doesn't.