Commentary and Analysis of “Building a Culture of Experimentation” by Stephan Tomke (HBR.org)
Tomke highlights how Booking.com fosters a culture of experimentation, allowing anyone in the company to test ideas without needing approval from a supervisor. He also draws parallels with companies like Expedia, emphasizing the significant insights gained from such a mindset. Despite Booking.com and Expedia being digital-first, the author argues that even non-digital companies can benefit from experimentation to uncover key consumer touchpoints. He then questions why, given its proven value, more companies aren’t engaging in experimentation, attributing this reluctance to cultural and behavioral barriers. His guidelines for overcoming these issues include cultivating curiosity, where failure is seen as an opportunity to learn, prioritizing data over opinions (especially those of the highest-paid executives), democratizing experimentation while streamlining the process, and maintaining ethical sensitivity in testing. Leaders, while potentially threatened, can still contribute by setting strategic goals and embracing system modernization, ultimately asking, "How willing are you to challenge your own assumptions?"
The article is supported by evidence showing that companies using A/B testing see performance improvements of 30% to 100% within a year of adoption ("Experimentation and Start-Up Performance" - Koning, Hasan, Chatterji), which underscores the benefits of testing in driving business outcomes. Moreover, A/B testing can be extended to areas of untapped potential. For instance, the study “Using Experiments in Corporate Strategy Research” (Croson, Anand, Agarwal) illustrates how experimentation can lead to valuable insights in corporate strategy, a field where such methods have traditionally been underutilized. The practical, evidence-based approach makes a strong case for wider adoption of experimentation across industries, suggesting it can drive more strategic innovation beyond digital-native firms.
While the author emphasizes the need for a cultural shift, they overlook the importance of technical skills and the adaptation of management teams. For instance, the article doesn’t address the significant challenges posed by data privacy regulations like GDPR. Before regulations like GDPR and CCPA, companies could rely on Cookie IDs and Device IDs for randomization in digital experiments, without needing to collect first-party data (opted-in, privacy-compliant data). However, these new laws require more effort and budget to gather this data. Large companies like Booking.com and Google can manage this, but many others struggle. Research from Blind, Niebel, and Rammer shows GDPR caused a 0.9% drop in product innovation post-enforcement. It also forced firms to reorganize their data management, which, while challenging, offers opportunities to use data more intelligently for product development.
The concepts discussed in class and the assigned readings reveal some parallels. For example, the idea of massive online experimentation aligns with Dr. Milkman's proposal for megastudies. Both approaches share a common goal: fostering a culture of experimentation rather than discouraging it. However, I find it challenging to understand how online experiments can maintain the same level of control as those guided by Induced-value theory, especially when considering confounding variables and other factors that might affect the outcomes. While the end goals are similar, the methodological rigor and control mechanisms differ significantly between these online experiments and more traditional, controlled studies.
Are online or digital experiments conducted at the scale mentioned in the article consistent with Induced-Value Theory for generating behavioral insights, or are they merely a way to make rapid decisions without seeking depth?
Are we facing significant data survivorship bias in these massive A/B experiments by not asking participants to provide additional information that could offer insights beyond their online interactions?
References:
"Experimentation and Start-Up Performance" - Koning, Hasan, Chatterji
“Using Experiments in Corporate Strategy Research” (Croson, Anand, Agarwal)
“The impact of the EU General data protection regulation on product innovation” (Blind, Niebel, and Rammer)
“So you want to run an experiment, now what?” (List, Sadoff, Wagner)
