Two +10% winners aren't +20%. Enter how many tests you run, how often they win and by how much, and see the real business impact of your testing program in revenue and ROI.
How to estimate
01Your store
Currency
02Your testing program
6 of 24 tests win
As reported by your winning tests, on revenue per visitor or purchase conversion.
03Costs
Typical test power
Overlap between winners
Winner's curse
False winners
Realistic extra revenue · first 12 months
$233k
Then up to $623k a year once this year's winners are all live.
Winners' lifts added up
$864k
Winners per year
6
Realistic lift per winner
+5.0%
Monthly revenue, first year
A winner reporting +6.0% is worth about +5.0% on average. Results that just pass significance overstate their lift by about 12%. At a 25% win rate, about 7% of winners are flukes.
From reported to realistic
Winners' lifts added up
$864k
Wins compound but overlap
−$94k$770k
Winner's curse: winning lifts are overstated
−$95k$676k
False winners: some are flukes
−$53k$623k
Realistic estimate
−$390k$233k
in the first 12 months, as winners arrive through the year
Return in the first year
ROI, year 1
87%
Runs entirely in your browser. Nothing you enter is sent to our servers.
Stop estimating what your winners are worth
Metricuno runs your A/B tests on real purchase data and adds up what each winner is worth per month, including what you miss while a winner waits to be built in.
Add up the lifts from a year of winning tests and you usually get a number nobody sees in the revenue graph. That isn't bad luck. Four things shrink every win between the test report and your bank account, and this calculator handles each one explicitly.
A +8% lift on a mobile product page is +8% on the revenue that goes through mobile product pages, not on all revenue. The quick estimate has one setting for this under Advanced; the test-by-test mode sets Revenue it touches per test. Upstream metrics need a second discount: an add-to-cart lift rarely turns into the same purchase lift, so Reaches revenue passes on only part of it.
Wins on different funnel steps multiply, because each step passes more shoppers to the next:
Wins on the same page are different. A clearer price and a bigger buy button can both fix “shoppers can't see what to do”, so the second one adds less. The overlap setting keeps the biggest win whole and counts only part of each further win on the same page.
Tests that only just reach statistical significance overstate their lift: among many tests, the ones that happened to overshoot are the ones that cross the line. The calculator shrinks each lift towards zero in proportion to its noise, using the size of real effects you expect. A precise result barely moves; a borderline one can lose half its lift.
A p-value below 0.05 doesn't mean a 95% chance the win is real. That chance depends on how often your ideas work at all. With the settings you choose, each win gets a probability of being real, and its value is weighted by it. The table also shows a Benjamini–Hochberg q-value for every win, counting all the tests you ran, including losers: a win with a q-value above your target doesn't survive a false discovery rate correction.
Here l is the reported log lift, s its standard error (from the p-value, interval or raw counts), π the share of ideas that really work and τ the typical size of real effects. It is a standard spike-and-slab model: a share of ideas do nothing, the rest have effects spread around zero.
A winner earns nothing until it ships to everyone, and many effects fade as novelty wears off and the site changes around them. Set the months until it's live and a half-life, and the calculator counts only the revenue-months that are left in your horizon.
The quick estimate needs only three numbers about your program, and works out the corrections from them. The share of winners that are flukes follows from your win rate: if a share π of ideas really work and tests have 80% power, the win rate is π × 0.8 plus the 2.5% of useless ideas that win by chance. The lower your win rate, the larger the share of flukes. The winner's curse follows from power: significant results from 80%-power tests overstate the lift by about 12%, from 50%-power tests by about 40%. Winners arrive through the year, so the first year earns only part of their full value; the “a year once live” figure is the full value.
In the test-by-test mode, the headline is the expected value and the range comes from 10,000 simulations that each decide which winners are real and how big they are, then combine them with the same compounding and overlap rules. Costs cover every test you ran (not just the winners), the hours to build the winners in, tools and anything else, so you see the return of the whole program.
Planning the next test instead? The A/B test calculator works out the sample size and test duration, and checks a finished result for significance. For a wider view, see the guides on experimentation programs and A/B testing.
Multiply the lift by the revenue that passes through what you tested, not by total revenue, and by the months the change will be live. Then correct for the winner's curse and the chance the result is a false positive, and subtract what the tests cost. This calculator does all of that and shows each step, in revenue, profit and ROI.
Combine all winners instead of adding their lifts, count every test you ran (losers too) in the costs and the false discovery correction, and report a range rather than one number. The calculator's waterfall shows stakeholders exactly how the reported lifts become a realistic figure.
Because lifts multiply and overlap. Two +10% wins on different funnel steps make +21%, not +20%. Two +10% wins on the same page often fix the same problem, so together they are worth less than +21%. Adding lifts also ignores that each test only touched part of your revenue.
Among tests that just reach significance, the reported lift is usually bigger than the true one, because the tests that happened to overshoot are the ones that crossed the line. The calculator shrinks each lift towards zero, more for noisy results and less for strong ones.
The share of your 'significant' winners that are actually false positives. It depends on how often your ideas really work: if few do, a large share of results under p < 0.05 are flukes. The calculator gives each win a chance of being real and a Benjamini–Hochberg q-value across every test you ran.
Your best estimate of the share of tested ideas that truly change revenue, which is lower than your share of significant results. If you're unsure, try a few values: the gap between them shows how much your conclusions depend on it.
Yes, in 'tests run in total'. They count towards the false discovery rate correction and towards the cost of the program, because you paid to run them too.
No. The calculator runs in your browser. Your numbers are saved in your own browser, and share links carry them in the part of the URL that browsers never send to a server.