Stefan Thomke: Why experimentation is key to finding success

A business only grows if it innovates. And innovation comes from evolving and applying fresh ideas.

Yet this move to modernization often carries peril: fundamental assumptions may be wrong, customers may not accept your ideas and new products may not work.

Many experts believe experimentation holds the key.

Booking.com’s openness to trial and error is often attached to its meteoric climb to becoming an everyday travel planning behemoth. The company today can conduct 1,000 concurrent experiments across 75 countries and 43 languages in under an hour.

From day one, a culture based on employees having the power to experiment without regulations is said to be the unequivocal catalyst for such impressive numbers and success.

Concisely put, the effects of this decentralization mean two major things. The vast volume of experimentation is big enough to provide compounding positive outcomes and, at the same time, gives numerous chances to investigate failure.

"For every online experiment that succeeds, nearly 10 don’t – and in the eyes of many organizations that emphasize efficiency, predictability and ‘winning’, those failures are wasteful."

"Controlled experimentation has revolutionized the way all companies operate their businesses and how managers make decisions," says Harvard Business School Professor and author Stefan Thomke, who has been researching and writing about the topic for more than 20 years.

"But most tests of new business initiatives are too informal. They are not based on proven scientific and statistical methods."

And the result? "Executives end up misinterpreting statistical noise as causation and making bad decisions," he says.

"Companies should follow a clear set of principles. In an ideal experiment, testers separate an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant."

In his 2020 book, Experimentation Works: The Surprising Power of Business Experiments, Thomke not only outlines why it works and the framework contributing to why, but also defines the attributes of a culture invested in it.

Most importantly, though, he understands how to diagnose and address cultural obstacles.

"Shared behaviors, beliefs and values are often an obstacle to running more experiments in companies. For every online experiment that succeeds, nearly 10 don’t – and in the eyes of many organizations that emphasize efficiency, predictability and ‘winning’, those failures are wasteful," he says.

Cultivating curiosity

So, how can organizations adopt an efficacious ethos of experimentation?

Thomke promotes the value of surprises and affirms that putting a dollar figure on unwelcome results is counterproductive. Furthermore, failures should not be seen as mistakes but as opportunities.

"Many organizations are too conservative about the nature and amount of experimentation."

"When firms adopt this mindset, curiosity will prevail. Many organizations are too conservative about the nature and amount of experimentation," he adds, noting how integral inquisitiveness is in this instance.

"Overemphasizing the importance of successful experiments may inadvertently encourage employees to focus on familiar solutions, or those that they already know will work, and avoid testing ideas that they fear might fail."

Data comes first

Whether based on ego, bias or prejudice, opinions are one of the culture of experimentation’s most significant obstructions.

Thomke has a very straight-faced way of seeing the intrusion of personal beliefs.

"The empirical results of experiments must prevail when they clash with strong opinions, no matter whose opinions they are. But this is rare among most firms for an understandable reason: human nature," he explains.

"We tend to happily accept good results that confirm our biases but challenge and thoroughly investigate bad results that go against our assumptions.

"The remedy is to implement the changes experiments validate, with few exceptions."

"Nothing stalls innovation faster than a so-called HiPPO – the highest-paid person’s opinion."

Getting top-ranking executives to abide by this rule is especially difficult, he says.

"But it’s vital that they do. Nothing stalls innovation faster than the so-called HiPPO – the highest-paid person’s opinion," he says.

However, according to Thomke, there has to be recognition of the right time and place. Not every scenario should be approached this way.

"I’m not saying all management decisions can or should be based on experiments," he says. "Some things are very difficult, if not impossible, to conduct tests on. For example, strategic calls on whether to acquire a company.

"But if everything that can be tested online is tested, experiments can become instrumental to management decisions and fuel healthy debates."

Diverse leadership

Various leadership models play a pivotal role in adaptability and success. There’s the advantages of inclusivity and the broader methodologies it encourages, as well as adopting agility enabling iterative approaches. But Thomke has three major points for leaders intent on experimentation and, by default, promoting its beneficial culture.

"Set a grand challenge that can be broken into testable hypotheses and key performance metrics. Employees need to see how their experiments support an overall strategic goal," he says.

"Leaders have to live by the same rules as everyone else and subject their own ideas to tests."

"Next, put in place systems, resources and organizational designs that allow for large-scale experimentation. Scientifically testing nearly every idea requires infrastructure: instrumentation, data pipelines and data scientists.

"Several third-party tools and services make it easy to try experiments, but to scale things up, senior leaders must tightly integrate the testing capability into company processes."

The third key is to be a role model, he says.

"Leaders have to live by the same rules as everyone else and subject their own ideas to tests," he says. "Bosses ought to display intellectual humility and be unafraid to admit, ‘I don’t know.’

"They should heed the advice of Francis Bacon, the forefather of the scientific method: ‘If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.’"