Experimentation Strategy
How to build the organisational muscle behind CRO — velocity, culture, hypothesis pipeline, and governance — so testing scales from a side project into a compounding growth engine.
Experimentation Strategy
The operating system around CRO — velocity, hypothesis pipeline, governance, and learning loops that scale testing from one-off wins into a compounding engine.
Experimentation strategy is the organisational layer that sits above any individual A/B test. It defines how many tests you run, where ideas come from, who decides what ships, and how learnings get stored so the next test starts smarter than the last. Most stores treat experimentation as a tactic — a tool the CRO specialist opens on Tuesday. A strategy treats it as a system with throughput, quality controls, and a feedback loop.
Done well, it turns a team that ships four tests a quarter into one that ships forty, with a higher win rate, because every test is informed by the last. Done badly, it produces a backlog of low-conviction ideas that ship slowly, contradict each other, and never get re-read.
If you're running a Shopify store between €1M and €15M, you've probably hit the same wall: one good test lifts checkout conversion 8%, everyone celebrates, and then the program stalls for six weeks while you argue about what to test next. That gap — between the first win and the fiftieth — is what a strategy solves.
There are four moving parts you have to design deliberately: how fast tests move through the pipeline, where hypotheses come from, who governs what ships, and how learnings get captured and re-used. Skip any one of them and the program plateaus. The rest of this page walks through each in the order they tend to break.
1. Velocity: the throughput that compounds wins
Experiment velocity is the number of valid tests you complete per unit of time — usually expressed as tests per quarter. It's the single strongest predictor of program ROI, because win rates cluster between 15% and 25% no matter how senior your CRO team is. If only one in five tests wins, running five tests a quarter gives you one winner; running twenty-five gives you five.
Velocity dies in three places: traffic constraints (you don't have enough volume to power concurrent tests), build constraints (every variant needs developer time), and decision constraints (tests sit in review for two weeks). The first you solve with sample-size planning and segment-level testing; the second with a no-code editor on top of your store; the third with a pre-agreed governance rubric so reviews take an hour, not a sprint.
2. Hypothesis pipeline and the culture behind it
A hypothesis pipeline is your backlog of test ideas, ranked by evidence and expected impact. Strong pipelines pull from four sources: analytics drop-offs, session replays and heatmaps, qualitative input (support tickets, NPS comments, user tests), and competitive teardowns. Hypothesis development is the discipline of converting a raw observation — "mobile checkout has a 42% drop on the shipping step" — into a testable claim with a predicted lift and a measurable outcome.
Pipeline volume is a function of experiment culture. If only the CRO specialist contributes ideas, you'll ship 20 hypotheses a quarter at best. If merchandisers, customer support, and performance marketers all submit through the same intake form, you'll see 80-plus — and the win rate climbs because frontline teams see friction that analytics misses. The cultural shift isn't "everyone runs tests"; it's "everyone surfaces evidence".
The most common failure: confusing activity for learning
Programs that brag about "50 tests this quarter" but can't tell you what they learned are running theatre, not experimentation. Velocity without a learning system produces contradictory results, repeated tests, and a team that loses faith in the data. Every shipped test should answer a question — and that answer should be searchable six months later.
3. Governance and learning systems
Experiment governance is the rulebook: minimum sample size, required pre-registration of primary metric, who can call a test early, what gets escalated to leadership. Without it, you get p-hacking by accident — someone peeks at day three, sees a 12% lift, ships the variant, and the lift evaporates in production. A one-page rubric (significance threshold, MDE, guardrail metrics, stop conditions) prevents 80% of these mistakes.
Learning systems are how you compound. Every concluded test — winner, loser, or inconclusive — goes into a tagged repository with the hypothesis, the result, the segment cuts, and one paragraph of "what this means for future tests". After two quarters this becomes the most valuable asset in your CRO stack: a searchable record that prevents you from re-running tests you already ran, and surfaces patterns ("urgency messaging works on apparel but not on supplements") that feed back into the hypothesis pipeline. This is also where growth loops emerge — wins in one funnel stage create input for the next.
How program maturity shifts velocity and win rate
Tests completed
Winning tests
Frequently asked questions
Running tests is a tactic; strategy is the system around it. Strategy defines how ideas get sourced, prioritised, reviewed, shipped, and re-read. A team without strategy ships a few tests a quarter and forgets the results; a team with one builds a compounding library of evidence that informs every future decision.
Twelve to twenty concluded tests per quarter is a realistic mature target at this revenue band. Below five and you're not generating enough learnings to compound; above thirty and you're probably running underpowered tests that won't replicate. Velocity matters, but only if each test hits minimum sample size.
Four sources: quantitative funnel drop-offs from GA4 or your analytics stack, qualitative signals (session replays, support tickets, customer interviews), competitive teardowns, and prior test results. A healthy pipeline pulls from all four — relying only on heatmaps produces shallow ideas, relying only on analytics misses the "why".
One named owner — usually a CRO specialist or Head of E-commerce — with cross-functional input. Distributed ownership without a single accountable person is the most reliable way to kill velocity, because no one chases the backlog or enforces governance. Contributors are many; the decision-maker is one.
Three signals: tests-per-quarter trending up, win rate stable or improving (not falling as you scale), and a learning repository the team actually opens before scoping new tests. Revenue lift is the outcome, but those three are the leading indicators that the system is healthy.
Around 30,000 monthly sessions on the page family you want to test is a reasonable floor for detecting 5-10% lifts at 95% confidence within two weeks. Below that, you can still test — but you'll need larger MDEs, longer runtimes, or segment pooling. Traffic shapes which questions you can answer, not whether you should run a program.
Two safeguards: a mutual-exclusion rule (no two tests on the same page element at once) and a learning repository that's searched before scoping. Contradictions usually mean the underlying behaviour shifted — different traffic mix, different season — which is itself a finding worth logging.
Yes — often more than winners. A loser tells you a hypothesis you believed was wrong, which prevents the team from spending the next quarter chasing variations of the same bad idea. The teams with the highest long-term ROI document losers as carefully as winners and tag them so they're findable.
Mainly in the hypothesis pipeline. AI is good at scanning analytics drop-offs and session replays to surface candidate hypotheses ranked by expected impact — the work that used to take a CRO specialist two days a week. It doesn't replace judgement on what to ship, but it lifts the floor on idea volume.
First wins typically land in weeks four to eight; compounding ROI shows up in months six to nine, once the learning repository is large enough to make future tests cheaper to scope. The first quarter is mostly investment — setting up governance, building the pipeline, and running the foundational tests on checkout and PDP.
Test ideas before you ship them
Run unlimited A/B tests, attach hypotheses to outcomes, and build a searchable archive of what works — and what doesn't.