ever-works/orgs/growth-experiment-design

Growth Experiment Design

An experiment exists to make one decision cheaper. If you cannot name the decision the result will change, you are not experimenting — you are collecting trivia.

Designing one experiment

1. Hypothesis. One sentence: "We believe [change] will cause [effect] for [audience] because [reason]." The "because" is mandatory — it's what you actually learn about when the result comes in.
2. Primary metric, singular. The one number that decides the outcome, defined precisely enough that two people would compute it identically. Secondary metrics may be watched; only the primary decides.
3. Written prediction. Before launch, record the expected direction and rough size of the effect, and what result would count as failure. Predictions made after seeing data are memories, not predictions.
4. Run length decided up front. Set the minimum duration or sample before launch, based on how much traffic the surface actually gets — and honor it. Data peeked at daily will eventually lie to you.
5. Smallest honest test. Prefer the cheapest version that genuinely tests the hypothesis: one landing page variant, one channel, one segment. A sprawling test that fails teaches nothing about which part failed.
6. Guardrails. Name what must not degrade while the experiment runs (brand voice, existing conversion paths, community trust). A win that breaks a guardrail is a loss with better numbers.

Prioritizing the backlog

• Score candidates on three axes: expected impact on the campaign's success measure, confidence in the hypothesis, and cost to run. Test high-impact, low-cost, genuinely-uncertain ideas first.
• An idea everyone is sure about is not worth testing; an idea nobody can connect to the funnel is not worth testing yet.
• Keep the backlog on one page, ranked; re-rank when a readout lands, since results change confidence scores.

Reading results

• Compare outcome to the written prediction first — that comparison is the learning, whatever the metric did.
• Classify honestly: clear win, clear loss, or inconclusive. Inconclusive is a legitimate result; stretching it into a win poisons the backlog's confidence scores.
• Every readout ends with a decision: adopt, revert, iterate with a named change, or stop this line of inquiry.

The campaign results readout

• One page per campaign: each experiment's hypothesis, prediction, result, decision; the funnel's actual shape versus the brief's assumption; a ranked list of what the next brief should change.
• Report negative results with the same prominence as wins — the readout's job is to make the next campaign smarter, not to make this one look good.