Continuous online experiments are the secret to developing a successful tech product
How do many successful tech companies develop their products and services? As Innovation Endeavors Principal, Davis Treybig shares in The Experimentation Gap, “companies like Airbnb, Booking.com, Microsoft, Google, Netflix, Doordash, and Stitch Fix run tens of thousands of experiments per year, and as a result are able to rigorously quantify the impact of most of the ideas and features they launch.”
The outcomes of these ongoing experiments can lead to incredible impact — from increased revenue to faster customer acquisition. But more importantly, a company that continually experiments is a company that is committed to iterating and building a far superior product that will evolve with its user’s tastes and needs. With that said, “most companies can not, despite their best efforts, realistically achieve an experimentation program anywhere close to what the aforementioned companies have,” shares Davis.
To this day (even though continual experimentation has been common practice for Amazon and others for several years) many companies remain dependent on manual processes, claim agility when their culture is anything but agile, and often only have the resources to run a handful of controlled experiments.
So, how does one break the cycle and embrace experimentation for their team? The following four core principles can help you structure a modern experimentation process — the same as the ones trusted by Netflix, Doordash, and Stitch Fix to name a few. And if you want a more detailed deep dive into the structure of a modern experimentation platform and advanced statistical techniques, head straight to Davis’s explanation.
Automation is essential
Any good experimentation workflow starts with end-to-end automation. If manual work is required for any step of an experimental workflow, then bottlenecks are likely to form within data science or data infrastructure teams.
Make sure you can self-serve
Self-service and simplified workflows matter just as much as infrastructure. Product managers, engineers, and designers must be empowered to create and manage their own experiments and learn from their outcomes. And of course, these experimentation workflows should simplify many of the statistical concepts of experimentation so this process is accessible to different individuals. Each department must feel empowered to run its own experiments in order for company-wide adoption to take place.
Invest in your DevOps
There’s a DevOps process to the experimentation model. This process should include upfront investment into monitoring experiments, debugging, ensuring that metadata about ongoing experiments is searchable and rich in information, and of course, that experiments have proper guardrails.
If you push for an experimentation model without investing in this DevOps management model, then your teams are sure to waste inordinate amounts of time on conflicting experiments, incorrectly configured experiments, poorly designed user experiences, and a host of other issues. Make sure you have guardrails (and a central team) in place to track, understand, and validate various experiments.
Socialize and collaborate
Social and collaboration features are critical! These underrated features for experimentation platforms foster learning, allow for wider team involvement, and will encourage the adoption of an experimentation process.
Without all four of these processes in place, your experimentation model is likely to result in chaos instead of success. If you’re ready to better understand the experimentation process including how to implement proven strategies used by industry experts, advanced statistical techniques, and how to develop an experimentation culture, click here to read more.