How to use Statistical Interpretation
Statistical methodology tells you whether a test is significant. Statistical interpretation tells you whether it's true, useful, and worth shipping — here's how to read results without fooling yourself.
Statistical Interpretation
Reading A/B test results correctly — separating real lift from random variance and knowing when a result is actually decision-grade.
Statistical interpretation is the judgment layer that sits on top of test methodology. Methodology tells you the maths checks out — sample size hit, p-value crossed, confidence interval calculated. Interpretation tells you what the result actually means for your store: is the lift real, is it big enough to ship, is it likely to hold once you roll it out to 100% of traffic?
It's the difference between a checkout test that 'wins' with a 4.2% lift and a checkout change that adds €180k of annual revenue. Same number, two different conversations. Good interpretation is mostly about resisting the pull of a clean-looking result and asking what else could explain it.
Most CRO programmes don't fail at the test setup. They fail at the readout — at the moment someone stares at a dashboard and decides whether the variant is a winner. That decision is where bad calls compound into bad roadmaps.
This guide covers the four interpretation skills that separate a useful experimentation programme from theatre: telling signal from noise, knowing when to call a test, recognising the common misreads, and applying judgment on top of the numbers. It's the practical companion to broader experiment analysis.
Separating signal from noise
Every A/B test result has two components: the real underlying difference between variants, and the random variance from which visitors landed in which bucket. Your job is to estimate how much of the observed lift is signal.
The cleanest tool for this is the confidence interval, not the point estimate. A test that reads '+8.3% lift' might have a 95% interval of [+1.1%, +15.5%]. The true effect is somewhere in that range, and the range itself tells you how much you actually know.
If your interval is wide and barely clears zero, you have a directional hint, not a decision. If it's tight and well above your minimum detectable effect, you have something shippable. Treat the width of the interval as a confidence signal in its own right.
The point estimate lies more than the interval
When teams report a single number — '+8.3%' — they're showing the most optimistic reading of the data. Roughly half the time the true effect is below that number. Always pair the headline lift with its confidence interval, or you'll consistently overestimate what you've shipped.
Knowing when to call a test
The single most expensive habit in DTC experimentation is peeking — watching a test live and calling it the moment significance flickers green. Early significance is mostly noise. Conversion rates wander a lot in the first few thousand sessions and stabilise as sample grows.
Set your sample size and runtime up front, then leave the test alone until both are hit. A Shopify apparel store running at 30k sessions a week typically needs two full weeks minimum to absorb day-of-week and weekend-vs-weekday traffic mix.
How observed lift stabilises as sample grows
Variant A (true lift: +3%)
Variant B (true lift: 0%)
Notice how a genuinely flat variant can read +9% lift at 1k sessions and a real winner can briefly look like a +12% blockbuster. Both converge toward the truth only after meaningful sample. Calling either test at day three would be a coin flip dressed up as analysis.
Common interpretation mistakes
Even well-run tests get misread. The same handful of mistakes show up across teams, and they tend to skew results in the optimistic direction — which is why programmes accumulate 'winners' that never compound into reported revenue.
The table below catalogues the misreads we see most often during experiment analysis reviews, and what each one actually costs you when it slips into your roadmap.
Common misreads, what they look like, and the real-world cost
| Misread | What it looks like | What's actually going on | Typical impact on reported lift |
|---|---|---|---|
| Peeking | Calling the test the first day p < 0.05 | False positive rate balloons from 5% to 20-30% | Overstated by 2-4x |
| Ignoring CI width | Reporting '+9% lift' with CI [-2%, +20%] | Real effect could be negative | Direction itself unreliable |
| Segment fishing | Finding a winning segment after the fact | Multiple comparisons inflate false positives | Most 'segment wins' don't replicate |
| Novelty effect | Big lift in week 1, fades in week 3 | Returning visitors react to change, not value | 30-60% of week-1 lift evaporates |
| SRM ignored | 50.4% / 49.6% traffic split treated as fine | Sample ratio mismatch signals broken assignment | Entire test invalid |
| Wrong primary metric | Click-through up, revenue flat or down | Optimised a proxy, not the outcome | Net-negative ship disguised as win |
Sample ratio mismatch deserves a special mention. If your 50/50 split is actually arriving as 51/49, something is wrong with assignment — bot traffic, caching, a redirect race condition — and the whole test is suspect. Always check the ratio before you read the result.
The judgment layer
Once the maths is clean, interpretation becomes a judgment call. A statistically significant +1.5% lift on a checkout test is worth shipping. A significant +1.5% lift on a homepage hero, on a small audience, with a wide CI, probably isn't — the implementation cost outweighs the expected value.
Good interpreters also ask whether the result makes sense. Did the variant change behaviour in a way the hypothesis predicted, or did it win for an unrelated reason? A button-colour test that lifts revenue 6% is almost certainly noise or a confound. A trust-badge test that lifts checkout completion 3% with consistent funnel behaviour is plausibly real.
The two questions to ask before shipping any winner
First: is the lower bound of the confidence interval still worth shipping? If yes, you have a robust win. Second: does the funnel behaviour match the hypothesis? If the variant won but the mechanism doesn't make sense, hold the ship and investigate — you've probably found a confound, not an improvement.
Frequently asked questions
Significance is a maths output — did the p-value cross your threshold given the sample size. Interpretation is the human judgment on top: is the effect big enough, plausible enough, and stable enough to ship? A test can be significant and still be a bad decision.
Generally no, unless you're using a sequential testing method built for it (like always-valid p-values). With fixed-horizon tests, early significance is mostly noise and your false positive rate can hit 20-30% instead of the nominal 5%.
Width depends on sample size and baseline conversion rate. A typical Shopify store running a checkout test for two weeks at 5% baseline conversion will see CIs roughly ±2-3 percentage points of relative lift. If yours is wider than ±5%, you're under-powered.
Treat the secondary win as a hypothesis for the next test, not a result from this one. Reporting a secondary winner as if you'd planned to measure it inflates false positives massively. Pre-register your primary metric and stick to it.
Returning visitors react to anything new — sometimes positively, sometimes negatively — independent of the variant's actual value. Detect it by segmenting first-time vs returning visitors and watching the lift over time. If the effect fades across weeks two and three, you had novelty, not improvement.
Yes, as a directional read for prioritisation. If a test points positive with a wide CI, it's worth retesting with more power. But you shouldn't ship from it — you don't know whether the true effect is +5% or -2%.
Hit your pre-calculated sample size AND run for at least one full business cycle — usually two weeks for most stores, to absorb weekday/weekend mix and weekly purchase rhythms. Whichever is longer wins.
SRM is when your traffic split diverges from the assignment ratio you set — for example, 50/50 arriving as 51.2/48.8 with thousands of sessions. It signals broken randomisation, bot contamination, or a redirect issue. Any test with SRM is invalid; fix the cause and rerun.
Be very skeptical. Post-hoc segment analysis inflates false positive rates because you're running many implicit tests. Use segment results as hypotheses for follow-up tests targeted at that segment from the start, not as ship decisions.
Experiment analysis is the full readout: data quality checks, segment views, learnings, next-test planning. Statistical interpretation is the specific skill of deciding whether the headline result is real and decision-grade. You can't do the broader analysis well without it.
Test ideas before you ship them
Run unlimited A/B tests, attach hypotheses to outcomes, and build a searchable archive of what works — and what doesn't.