Why do Experiments Fail? 30 Learnings
After running 100+ experiments at Miro
🤗 Who is this guide for?
Founders / CEOs
Product Leaders
Growth Product Leaders
Growth Marketing Leaders
Growth Design Leaders
Growth Analytics Leaders
Growth Engineering Leaders
❤️ Why should you care?
Easy! Learn from my mistakes at Miro so you can set yourself and your team up for success.
✅ What do you get?
The guide has five sections:
Target Metrics (5 learnings)
Experimentation Portfolio (10 learnings)
Team (3 learnings)
Leadership Buy-In (2 learnings)
🎁 Bonus Tips (Best for Last) (10 learnings)
😑 Before we start!
Experimentation is a team sport; you and your team are playing it together.
Now, we are ready.
🏁 Let's start!
Target Metrics
Your target metric represents a complex action!
Break the metric down! Otherwise, with a single experiment, you won't be able to move it. In Miro's case, the core actions revolve around collaboration. Not a walk in the park.
Your target metric is lagging!
The impact on the metric can be observed only in X weeks. Activation (for example). Focus on leading indicators! But only those that you proved to be correlated with the target metric. Don't cheat,
Your target metric is a business metric not aligned with the users' problem!
Iterate on the target metric! Be aware that you will need strong leadership buy-in to make this happen.
Your experiments cannibalize other teams' target metrics!
Calculate the impact on a higher-level business metric (Engaged Users, ARR, etc.)! Groom your relationships to work such cases effectively.
Your success is perceived only as impact on the target metric!
Change your success criteria! Success = High value and actionable learnings, whether the hypotheses were proved or disproved.
🟢 Work with me!
There are three ways we can work together. Check those out. 😉
Experimentation Portfolio
You are running only 'quick win' experiments!
Mix it up with big bets! Otherwise, you limit your impact, leading to a long-term lack of leadership buy-in.
You are running big bets backed by low-confidence hypotheses!
Sequence the experiments! Run small experiments to validate your bigger hypotheses.
You are spread on too many opportunities and jumping around!
Focus and iterate! Experiments rarely work on the first attempt. Post-analysis and iteration are key to making it work.
You experiment everything!
Not everything should be an experiment! If cost > return and/or potential learnings are of low value, don't experiment. Experimentation culture doesn't obsolete logic and sense.
You are looking for the magic framework to prioritize experiments!
Don't overcomplicate it! Considering the impact, confidence, cost/complexity, and sequence should be sufficient.
You are an estimation optimist!
Although people are skeptics in life, they become optimists when it comes to estimations. Discount your experiments! Genuinely evaluate your hypothesis confidence.
You set OKRs on the number of experiments or the experiments' success rate!
Stop! It diverts the team from focusing on impact and learning. Don't do it unless you are just setting up the team and want to get some quick wins going.
The growth forecast (and OKRs) is not aligned with the estimated team experimentation portfolio impact!
Go back to the drawing board! Both in terms of forecast and expectations from your team. Work through the gap before signing the Q off.
You slack on post-analysis.
Brrr... Now I am getting angry! Post-analysis is the most important part of experimentation. The quality of post-analysis determines the teams' success! For that, teams should be adequately staffed. A Dedicated Product Analyst!
Your experiments' priorities are driven by infrastructure debt!
Can't properly segment or have clashing experiments. Make a business case to prioritize experimentation infrastructure work! Your infra must evolve in tandem with your teams' experimentation skills.
Team
Your team setup is incomplete (baby team)!
APED is your MVP! PMs should not 'do it all'.
A - Analytics
P - Product
E - Engineering
D - Design
Your team setup is incomplete (mature team)!
For mature teams, aim for (AMPED+R) team structure!
A - Analyst
M - Marketing
P - Product
E - Engineering
D - Design
R - Research
You have a hiring quality gap!
Hiring for growth teams is different from hiring non-growth product teams. This applies to both Product, Design, Engineering, and Analytics. How different? Reach out, and I will share.
Leadership Buy-In
You have an executive buy-in problem!
Experimentation is not less people business than numbers business! Over-communicate research insights, experiments learnings, your process and experiments sequence, and spin up cross-company ideations sessions.
You have a cross-company stakeholders' buy-in problem!
Make teams around you successful! Enable other teams with your learnings. Help cross-functional team to achieve their OKRs. The best recipe for LOVE is mutual success.
🎁 Bonus Tips (Best for Last)
Reforge is mandatory but not sufficient!
Reforge thinking should guide your thinking. However, it's not a magic bullet to solve all your problems. Don't do something that doesn't make sense to you just because Reforge says so.
(Growth team) A twelve-month experimentation roadmap is not a thing!
Detailed long-term roadmaps contradict Growth DNA. Three months of experiments portfolio planning should be your roof-shot time horizon.
The Bayesian approach is not a curse word!
Don't limit your experimentation stack! There are pros and cons to the Bayesian approach. Bayesian is flexible, especially for small improvements, and valuable when running many experiments. Be smart about it!
Monitor running experiments!
Don't just leave them alone and sad. Monitor, not for concluding the experiments early on but to detect potential errors and save time. Both PMs, Analytics, and Engineers should monitor running experiments. Each has an angle.
Gradually push the limits on the complexity of experiments you run!
Focus on optimizing the time it takes to set the experiments up and decreasing the error level. Experimenting with business model changes is the hardest!
Seasonality and poor traffic should not be your go-to excuse for not hitting targets!
Those should be already embedded in the forecast.
Research is gold, but don't research for the sake of research!
First, leverage existing company-wide research outcomes. Research is a $ investment. Procrastination by Research is a thing.
Always know what's next!
You should always be ready to answer what you plan to do next per each different future outcome of an experiment.
PM / Analytics relationships tend to be overlooked!
Nothing can break the trust between PMs and Product Analytics folks more than PMs finding (not on a one-time basis) mistakes in PA's work.
Experimentation is awesome!
The learnings you bring to the table can fundamentally change the company’s strategy and influence other team roadmaps and decisions.
You can also check out the slick PDF version of this write-up. 👇
