“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day” Founder and CEO of Amazon, Jeff Bezos
The purpose of this article is to eliminate the babel tower of innovation mumbo-jumbo, and to shed light on the only principle that really matters when creating new products, services and startups, namely the principle of experimentation.
I will explain why experimentation is the beating heart of all the modern innovation methodologies such as Design Thinking, Lean Startup, Pretotyping, and Design Sprint.
And we’ll take a look at just how important experimentation is to the success of the world’s most successful companies such as Amazon, Facebook, and Google.
Last, I will give a few tips on how you can get started introducing experiments into your own organisation.
The many faces of experimentation
In a trial-and-error world, the winners are the ones who can validate their ideas the fastest. And whenever we are faced with great uncertainty, as we most often are when creating innovation, trial-and-error experiments are the best we have.
We use these experiments to mitigate risk and to increase our knowledge by testing our hypotheses, validating which are true and which are false, adjusting course when needed in order to reach customer/solution fit and later product/market fit.
This process of systematically falsifying our key assumption is the very essence of innovation.
But we have so many names for it that it’s easy to get confused.
Some people call this process “fail fast”, at IDEO they call it Design Thinking, Eric Ries calls it “Lean Startup”, Dave Thomas and other developers call it “Agile”, and at Google they call it “Pretotyping” or “Design Sprint”.
Whatever you call it, the point is that experimentation is the common denominator across all popular innovation methodologies, and as such the only principle that really matters.
Now, you might say: This sounds reasonable, but how do we know for sure that experimentation is at the center of all these innovation methodologies?
Let’s ask them (them being our most popular innovation methodologies).
Me: Yo, innovation methodologies!
Innovation methodologies: Sup..?
Me: What is the key principle of innovation?
Innovation methodologies: It depends man… I come with a multitude of different views. Are you a startup or a corporate? Do you see a problem and have a proposed solution for it, or are you still in ideation mode? Like I said, it depends…
Me: Uhmm, that is pretty useless… I’m looking for quick-fix, instagram-friendly answers, get it?
Innovation methodologies: Look, I’m kind of busy selling books about innovation and coming up with new names for the same thing. Why don’t you do it yourself?
Me: Fuck it, I might as well.
Now, if the collective body of our most popular innovation methodologies could speak with a unified voice, the instagram-friendly answer we are looking for, would probably be something like the one, I found when writing my previous article, “Fast Track 🏎️ 💨 to learning Design Thinking, Lean Startup, Agile, Pretotyping, and Design Sprint”.
After analysing and working with Design Thinking, Lean Startup, Agile, Pretotyping, and Design Sprint, I found that while the application of each framework varies, the one principle that they all have in common, is the principle of experimentation.
- Design Thinking calls it “Prototyping”
- Lean Startup calls it “MVP”
- Agile calls it “Sprint”
- Pretotyping calls it, well… “Pretotyping”
- And as Design Sprint is heavily inspired by Design Thinking, they too call it “Prototyping”
While the names might differ, the process is almost identical. And at the core of that proces we find experimentation in every framework.
So, if you really want to be a proficient innovator, it makes a lot more sense that you study, practice, and truly understand how experimentation should be implemented on a large scale in your organisation, and less if the process is called Design Thinking, Lean Startup, or something else.
Now, with that out of the way, let’s turn our attention to what the most successful companies in the world, have to say about experimentation.
Quantity beats quality; How Amazon, Facebook, and Google, are using experiments to be successful
These days, the true test of how innovative your company is comes down to how well it experiments.
Due to a combination of new technology and the growing popularity of innovation methodologies such as Design Thinking, Lean Startup, Pretotyping, and Design Sprints, the pace of experimentations are increasing rapidly in both startups and in corporate organisations.
Here is a brief look at how many experiments some of the world’s most innovative companies are carrying out on a yearly basis:
The sheer number of experiments carried out per year should make most innovators start sweating bullets!
You: Alright… I see what you are getting at. Experimentation seems pretty important. But how do my company rev our experimentation engine to have such an astonishing output?
Me: “Good question!
While there is no quick-fix solution for transforming an organisation to embracing experimentation at all levels, we can find insightful inspiration when studying the world’s most successful companies such as Amazon, Facebook, and Google.
While these companies have almost become anecdotal to study, I find it truly remarkable how explicit they are about experimentation as the core principle leading them to success.
Amazon 🗺️ 🛒
Jeff Bezos is notorious for his experimentation strategy and has introduced numerous successful inventions to the world such as AWS, Kindle, Alexa, Amazon Prime, and much more.
In a letter to his shareholders, he explains how deeply rooted experimentation is in the Amazon culture:
I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.
Founder and CEO Amazon, Jeff Bezos
In a 2011 interview, he further explained his thoughts on experimentation:
“We’ve tried to reduce the cost of doing experiments so that we can do more of them … If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.”
Another time, Jeff Bezos stated that experimentation is not part of the innovation strategy, it is the innovation strategy:
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day”
When success is defined as a function of the number of experiments you run, by a guy who is soon to become the only store owner on the face of earth, I pay attention!
Facebook has spend a lot of time and effort in perfecting their testing framework that allows every engineer to perform small scale experiments on new features and ideas.
In a recent interview, Mark Zuckerberg explained how he views the overall success of Facebook relating to experimentation:
“One of the things, I’m most proud of that is really key to our success is this testing framework … At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.”
This testing framework creates a mind-blowing number of experiments throughout the year eclipsing +100.000, and so the Facebook platform has a good chance of staying ahead of competition, even if they currently find themself in a world of trouble.
At Google’s innovation lab, Google X — the CEO, or should I say, the Captain of Moonshots, as Astro Teller calls himself, is making a habit out of promoting people who fail fast.
That is, the people who are given a team, money, and a timeframe, and are able to prove that the project they are running is no longer worth pursuing.
That may sound crazy that you would promote failure, but it makes good economical sense, as the money saves from not continuing running experiments for another year with a full team, is counted in the millions of dollars.
Also, Astro Teller makes no secret about the fact that he is doing it to ensure that he forsters a culture at X, where experimentation and failure is an expected part of the process. So much so, that you get promoted for failing fast.
What happens when you don’t experiment?
You: Alright, I get it — experiments are important — but what happens if you don’t experiment?
Me: Good question!
Let’s go back to owner of the “everything store” founder and CEO of Amazon, Jeff Bezos, as he explains example what happens when companies don’t experiment.
Some decisions are consequential and irreversible or nearly irreversible — one-way doors — and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don’t like what you see on the other side, you can’t get back to where you were before. We can call these Type 1 decisions.
But most decisions aren’t like that — they are changeable, reversible — they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high-judgment individuals or small groups.
As organizations get larger, there seems to be a tendency to use the heavy-weight Type 1 decision-making process on most decisions, including many Type 2 decisions. The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention.
Spot on Mr. Bezos 👏
So it seems that our ability as an organisation to perform experiments is the core driver behind successful innovation.
And that is the principle of experimentation.
Now, it’s your turn
While there is no easy-fix for revving up your experimentation engine at your own company, there definitely some tips that can get you started on the right path:
- Work toward building a platform or approach that makes it possible to run not just twice as many experiments as you already do but 10 to 100 times more.
- Empower everyone, also your most junior employees to conceive and perform their own experiments. This can be done by handing out pre-paid debit cards with a $1000 limit for experimentation, along side an instruction kit (this is what Adobe is doing with Kickbox).
- Explain and teach the process of experimentation to all members of the organisation. If you are in doubt which methodology to choose, read more here.
- Come up with a standard framework for validating results.
- Study and learn the various types of experimentation design that are available in the literature (or wait for my next article that will do exactly that).
- Share the results and lessons learned from experiments across the company in a experimentation document, and use that to inform current and new employees about what have previously been tested and how it performed. Include both successful- and failed experiments.
- Set aside a percentage of your overall budget strictly for experimentation. Most companies spend 0%–5% of their time and money on this, when it should be closer to 10%-20%.
In my upcoming article I will give examples of the various test designs that are available, and how you can use them to validate your most critical assumptions with speed.
Thanks for sticking around! I hope, you learned something valuable a long the way.
If you have any questions, comments, or good examples, please let me know!
I will be more than happy to update the article and to give credit to everyone helping out 😀