I recently came across a couple of good articles on business experimentation from my meanderings of more than thirty LinkedIn group discussions. Randomized experiments or related quasi-experimental designs for measurement are now part and parcel of evidence-based business. Indeed, whether you refer to the focus as data-driven, super-crunching or big data analytics, central are large data sets, statistics/machine learning algorithms and sophisticated designs.

Writing in the European Business Review, Harvard Business School professor Stefan Thomke derides what he sees as the current obsession with cost-side optimization and standardization – “Six Sigmaing” – that he feels may ultimately come back to haunt practicing companies. “Such organizations leave themselves vulnerable to underinvesting in experimentation and variation, which are the lifeblood of innovation … And it is only through such experimentation, which might include structured cause-and-effect tests, informal trial-and-error experiments, and rigorous randomized field trials, that companies can unlock their true capacity for innovation.”

The poster children for the innovation problem, according to Thomke, are 3M and now-departed CEO W. James McNerney Jr., who built a Six Sigma reputation at General Electric. Under McNerney’s leadership and process improvement prioritization, 3M started losing its innovation edge, new product revenue dropping from 33 to 25 percent of sales.

If Six Sigma’s preoccupied with the cost side of P&L, innovation and hence experimentation are more concerned with revenue. Thomke argues that in a downturn such as world economy has recently experienced, it’s important to step on the innovation accelerator. “… those companies that maintain their experimentation when business is slow will be all the more prepared when the market eventually picks up.”

For Thomke, innovation requires experimentation, since in the complex business world, passive observation and exploration are generally not enough to discern complex environments and linkages among variables. Properly designed and implemented experiments, though, can shed light on important causal relationships. For example, “randomized field trials can help companies determine whether specific changes (such as a new layout for a chain of retail stores) will lead to improved performance (a significant bump in sales).” And medical researchers can determine the efficacy of a new treatment by executing a trial that randomly assigns patients to treatment/control groups and comparing outcome measures.

Even properly-executed experiments – those with high internal validity – can still suffer from external validity shortcomings if the sample isn’t representative of the larger population or the findings don’t generalize beyond the specific experimental settings.

The good news for business innovators is that experimentation has never been cheaper. Many B2C companies design, implement and analyze thousands of experiments weekly, fueled by Internet access to customers and inexpensive technologies. Amazon and Google users participate in trials almost every time they log in.

The Thomke doctrine appears not to have been lost on 3M replacement CEO George Buckley, who countermanded many of his predecessor’s dictums, increasing the R&D budget and unshackling the research scientists, tellingly stating: “Invention is by its very nature a disorderly process ... You can’t put a Six Sigma process into that area and say, well, I’m getting behind on invention, so I’m going to schedule myself for three good ideas on Wednesday and two on Friday. That’s not how creativity works.”

Behavioral economist Dan Ariely is frustrated with business reluctance to experiment as well: “I’ve often tried to help companies do experiments, and usually I fail spectacularly,” he notes, citing personal unfulfilled consultations in marketing and HR.

Ariely offers two psychological explanations for organizational apprehension over experimentation. First, adoption of the experimental method mandates that business trade off short-term losses for the potential of longer-term gains. And that’s only the potential for gain. The experimental mentality means accepting that you’ll often fail – and of course retain management support in the process. In the what-have-you-done-for-me-this-quarter mentality, that accommodation may not fly.  

Second is the sense of security – or maybe just cognitive dissonance – that derives from having highly-paid experts make strategic choices. Historically, companies have been much more comfortable adopting the recommendations of experts over those of cold analytics. I suspect experts have been more comfortable with that situation as well!

But that’s changing. Some of the business consultants I’ve worked with in the past and watched pooh-pooh analytics are now on the bandwagon. The major strategy and business consultancies are aggressively establishing big data and analytics practices. And the notion that analytics trumps experts for many predictive decisions is almost universally accepted now, even if grudgingly by some. This is assuredly good news for business.

I’ll more to say on business experimentation in upcoming blogs. Happy Fourth of July!