How to write an Experiment Document

Writing helps you think

I’m a big proponent of writing things down. I believe that it brings clarity to your thinking and is a great way for others to help you find flaws in your logic. When I worked at Microsoft, we used to put together (very) long Product Specs that covered everything from the businesses case to the intricate details of how a feature should be implemented. While I believe that it’s often unnecessary to go to the level of granularity of interactions for every click (unless you work in a large or non-co-located team), the notion of writing down why you’re pitching the thing you’re pitching is highly valuable.

The likes of Jeff Bezos and Stephen Sinofsky are also huge supporters of getting their teams to spend the time to write.

While many startups believe that “documentation is not Agile,” I believe in the opposite. At bare minimum, I believe that everything that a Product Manager pitches should be paired with an Experiment Document. Here’s a basic outline.

1. Background & State of the World

Provide some context for why this work has bubbled up – how does it tie into the company’s current quarterly goals and OKRs?

2. Customer problem you’re solving for

Bring the customer into the story. Do you have quotes, customer usage patterns, or recordings that help solidify the customer problem?

3. Hypothesis

Write a clear hypothesis in the form of

  • We believe that by doing X
  • Y will happen
  • We know that we’re successful if we achieve metrics A, B, and C in T time

Speak to the potential impact to the company’s most important metric here. For example, if increasing usage by 30 minutes per day can be correlated with $X,000 of revenue, model that out.

4. Assumptions

This part is really important – what are the assumptions you’re making that led you to believe that this hypothesis is true? If this experiment fails, you need to jump straight to this section to identify which assumption was wrong and fold it into your learnings.

5. Past Data

Do you have historical data or past experiment results to justify why this work is important?

6. Experiment Design

Are you going to A/B test this design? To which cohort? How long is this test going to run for?

7. Next Steps

What are the next few experiments or decisions you’ll make if this experiment proves to be successful?

Do you have any other experiment doc formats you’d like to share? Do reach out to me at [email protected]. Would love to learn more about what you’re doing.

Yi-Wei Ang

Author Yi-Wei Ang

Director of Product @ TradeGecko | Product Coach @ 18g.io

More posts by Yi-Wei Ang