Experiment Culture

Experimentation strategy is the plan — which surfaces to test, what hypotheses to prioritise, how to allocate traffic. Experiment culture is the behaviour layer underneath it: whether the team will actually report a flat result honestly, and whether leadership will accept it. Strategy tells you what to do; culture decides whether the answer reaches the decision-maker intact.

HiPPO stands for Highest-Paid Person's Opinion. It matters because the single biggest predictor of a failing experiment culture is how often a senior opinion overrules a clean statistical result. One override teaches the team that data is decorative. Three overrides in a quarter ends the program — people stop bringing hypotheses that might embarrass the boss.

No. The moment a losing variant becomes a career liability, your designers and PMs will only propose safe tests they're confident will win — and safe tests teach you nothing. Tie performance reviews to test volume, hypothesis quality, and learning velocity instead. The outcome of any individual test should be irrelevant to the person who designed it.

15-25%. If your win rate is above 40%, you're almost certainly testing only changes you already believe will work — meaning the program is functioning as a confidence-bolstering ritual rather than a learning engine. A lower win rate paired with a higher kill rate is the signature of a team taking real swings.

Start by separating the two roles the founder plays: idea-generator and decision-maker. The founder can still propose half the hypotheses; they just can't override results on the ones that lose. Publish test results in a shared channel before any meeting happens, so the data lands before the politics do.

Renaming flat tests as 'directional wins' and shipping them anyway. This is the cultural equivalent of moving the goalposts mid-game. It quietly destroys the team's ability to trust their own results, because everyone learns that the bar for 'winning' is whatever leadership wanted that week.

Six to twelve months for the basics, two to three years for it to feel native. The leading indicator is when someone outside the growth team — a customer-service lead, a warehouse manager — proposes a test hypothesis unprompted. That's the moment experimentation stops being a department and starts being a habit.

A four-person team needs it more than a forty-person team. With fewer people, a single HiPPO override poisons a higher share of the roadmap. Small teams should formalise one rule on day one: results are reviewed before opinions are shared, every time.

Track four numbers quarterly: tests shipped, kill rate, share of hypotheses from outside leadership, and HiPPO overrides. Review them in a 20-minute retro, not a dashboard. The conversation is the measurement — the numbers just give it somewhere to start.

Shipping a losing variant because a senior stakeholder liked it, then telling the team the test 'wasn't conclusive enough'. Once the team sees that data loses to preference, hypothesis quality collapses within a quarter and you're back to running on opinions in under six months.