Managing your experiment backlog
One thing that has helped me tremendously is physically visualizing my experimentation backlog to help me prioritize the team’s work.
Here’s the general principle I abide by when I allocate resources towards multiple options. I first consider the potential Expected Value (EV) of doing something. In a simple definition:
Expected Value = Probability x Potential Upside
Plotting your Expected Value
Every experiment or work item has a potential EV. It would be useful to plot out your experiments in a 2 x 2 matrix. The x-axis being Probability of Success and the y-axis being Potential Return. The matrix looks something like this:
When you have plotted everything out, what you’ll find is the following:
Top right corner (High-Impact, High-Confidence)
- These are efforts that you’re highly bullish about. Not out of pure speculation, but because your past successes have given you confidence. Show these details in an Experiment Doc.
- You should be able to speak to what past experiments have led you to this conclusion
- These are the efforts you should focus on scaling up and putting in more resources on (these are where larger development costs start to make sense).
Top left corner (High-Impact, Low-Confidence)
- Think about the quickest and fastest ways to test these things so that you can prove/disprove the hypotheses related to these experiments.
- Once these experiments have been proven out, you will be able to move them to the top-right quadrant as your confidence increase
Bottom Left Corner (Low-Impact, Low-Confidence)
- Deprioritize these experiments until you get qualitative or quantatitative data that challenges (and increases) the potential impact of these experiments.
Bottom Right Corner (Low-Impact, High-Confidence)
- Don’t ever pick these things up – you’re confident that they don’t work!
Final note: Your return-on-investment depends on your cost
Add in lightweight t-shirt sizing estimates for each item. Within your top-right corner, you should think about which of these items you should prioritize relative to the cost.
Remember,
ROI = Impact(EV) / Cost
So make sure you have the highest EV to Cost ratio in the work that you do!
