How to use this A/B test calculator
The calculator covers both ends of a test. Before you launch, the Plan tab tells you how many visitors you need and how many weeks that takes at your traffic. After the test, the Analyze tab tells you whether the difference is real, how big it probably is, and whether you can trust the data at all.
Planning a test
Enter the weekly visitors who'll actually see the change, your current conversion rate, and the smallest lift that would be worth shipping (the minimum detectable effect, or MDE). The chart shows the trade-off: the longer you run, the smaller the lift you can detect. If the answer is “14 weeks”, test a bolder change rather than waiting. Halving the lift you want to detect roughly quadruples the traffic you need.
Don't know your traffic yet, or have a fixed window? Switch “Plan from” to A test length, drag the slider to 2, 4 or however many weeks you have, and the calculator tells you how many visitors a week the page needs. Then compare that with your analytics.
Add a start date and you get the end date, plus a warning if the test overlaps Black Friday, Christmas or Easter, when shoppers behave unlike the rest of the year.
Analyzing results
Enter visitors and conversions for each variant (or conversion rates, or paste the results table from your testing tool). You get a plain-language verdict, the lift with a confidence interval, the p-value, the Bayesian chance that each variant beats the control, and a sample ratio mismatch check. Add your average order value to see what the lift is worth per year.
Coming from the Plan tab, the calculator also knows your planned sample. If a variant looks like a winner before you've reached it, you get a warning instead of a green light: early “winners” are the most common way A/B tests mislead. When you're done, Copy summary gives you a ready-to-paste Slack update and Download image a shareable results card.
How the numbers are calculated
Significance uses a two-proportion z-test with a pooled standard error, the same test experiment analysis in Metricuno runs on live data:
The interval for the relative lift uses the delta method on the log of the rate ratio, which is why it's asymmetric: a lift can't fall below −100%, but it can exceed +100%. Sample size uses the standard two-proportion formula (pooled variance under the null, unpooled under the alternative):
Worked example: a 3% baseline, a +10% relative MDE (3.0% → 3.3%), 95% confidence and 80% power need about 53,200 visitors per variant. At 20,000 visitors a week that's six weeks. The Bayesian results use uniform Beta(1, 1) priors and 20,000 Monte Carlo draws from each variant's posterior. For more on choosing inputs, see sample size and confidence intervals.
Mistakes that make A/B test results lie
Peeking and stopping early
Checking every day and stopping the first time p dips below 0.05 can push your real false-positive rate well above 5%. Decide the sample size up front and read the result once you reach it. If you need to monitor continuously, use a sequential testing method designed for it.
Running for less than a full week
Weekend shoppers behave differently from weekday ones. A test that runs Monday to Thursday measures a different audience than your real traffic. Always run whole weeks, which is why the Plan tab rounds up.
Ignoring a broken split
If you planned 50/50 and got 47/53 on tens of thousands of visitors, the randomisation is broken, and so is the result. The SRM check flags this automatically.
Testing too many variants on too little traffic
Each variant splits your traffic further and adds another chance of a false winner. With limited traffic, one bold variant beats four timid ones. More pitfalls in common A/B testing mistakes.
Frequently asked questions
How long should I run an A/B test?
Until you reach the sample size you planned before starting, and always in full weeks so weekday and weekend behaviour are equally represented. Use the Plan tab: enter weekly visitors, your conversion rate and the smallest lift worth detecting, and it returns the number of weeks. Don't stop the moment a result turns significant; that inflates false positives.
What does statistically significant mean in an A/B test?
It means the difference you observed would be unlikely if the variants actually performed the same. At 95% confidence, a p-value below 0.05 is significant: if there were truly no difference, you'd see a gap this large less than 5% of the time. It does not mean there's a 95% chance the variant is better; that's what the Bayesian “chance to beat control” estimates. More in our guide to statistical significance.
Which confidence level should I use: 80%, 85%, 90%, 95% or 99%?
95% is the standard and the safe default. Lower levels need less traffic but accept more false winners: at 80% confidence, one in five tests with no real difference will still look significant. That can be a fair trade for cheap, easily reversed changes on low-traffic sites. Use 99% for changes that are costly or hard to undo, such as pricing.
Should I use a one-sided or two-sided test?
Use two-sided unless you decided in advance that you only care whether the variant is better and will treat 'worse' and 'no difference' the same way. One-sided tests need about 20% less traffic, but they can't tell you a variant is hurting conversions.
What's the difference between the frequentist and Bayesian results?
The frequentist p-value asks how surprising the data would be if there were no real difference. The Bayesian numbers answer the question most people actually have: how likely is it that B is better, and how much do I risk by shipping it? This calculator shows both side by side. With enough data they agree; when they don't, the test needs more traffic. See Bayesian testing for the longer version.
What is a sample ratio mismatch (SRM)?
An SRM is when the visitor counts per variant don't match the split you configured, for example 10,000 vs 11,000 on a 50/50 test. A chi-square test tells you whether the gap is too big to be chance. If it is, something is broken (redirects, bot filtering, caching, tracking only firing on one variant) and the result can't be trusted, however significant it looks.
Can I test more than one variant at once?
Yes, up to four variants plus the control. Each extra comparison raises the odds of a false winner, so the calculator applies a Holm correction to p-values in the analysis and splits alpha across comparisons in the plan. That's why A/B/C/D tests need noticeably more traffic than a simple A/B test.
Why don't you show observed power?
Observed (post-hoc) power is a direct function of the p-value, so it adds no new information and is easy to misread. Power belongs in planning: decide your sample size up front with the Plan tab. For a finished test that isn't significant, we show how many more visitors you'd need to confirm the observed lift instead.
Is my data stored anywhere?
Not on our servers. Every calculation runs in your browser, and your last inputs are remembered only in your own browser's local storage so you can pick up where you left off ('Start over' clears them). 'Copy link' puts the numbers after the # in the link: browsers never send that part to a server, and the page removes it from the address bar before any analytics loads.