I was preparing to start writing a blog entitled “Experimentation in Business” based on the terrific December 2014 Harvard Business Review article “The Discipline of Business Experimentation”, by Stefan Thomke and Jim Manzi, when my HBR email management tip of the day alerted me to a note “A Testable Idea is Better than a Good Idea”, by Michael Schrage.
Schrage argues that because testable hypotheses come with built-in accountability, they're superior to even outstanding business ideas not framed for disproof. “A business hypothesis suggests a possible or plausible causal relationship between a proposed action and an economically desirable outcome...Testable hypotheses seemed a faster and, frankly, better gateway to innovative action and active innovation. Testable hypotheses encourage and facilitate active experimentation and learning in ways that good ideas simply cannot.”
Thomke and Manzi build on Schrage's hypothesis theme to propose a methodology revolving on experimentation for both testing hypotheses and discerning the cause and effect linkages of business behavior if we do more of X then we should see more of Y. “In an ideal experiment the tester separates an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant, and then manipulates the former to study changes in the latter. The manipulation, followed by careful observation and analysis, yields insight into the relationships between cause and effect, which ideally can be applied to and tested in other settings.”
The authors identify questions that must be satisfactorily addressed by stakeholders before a business is ready to benefit from experimentation. “Does the experiment have a clear purpose? Have stakeholders made a commitment to abide by the results? Is the experiment doable? How can we ensure reliable results? Have we gotten the most value out of the experiment?”
I particularly like T&M's business description of the methodology process. While they note that the randomized experiment is the platinum standard for organizational learning, they also cite the potential of big (wide) data to both play a prominent role where randomization isn't feasible, and also to embellish experiments for the statistical analysis of cause and effect.
Finally, Thomke and Manzi offer a cautionary tale on the potential for negative unintended consequences introduced by experiments. “In many situations executives need to go beyond the direct effects of an initiative and investigate its ancillary effects...... A few years ago, Wawa, the convenience store chain in the mid-Atlantic United States, wanted to introduce a flatbread breakfast item that had done well in spot tests. But the initiative was killed before the launch, when a rigorous experiment—complete with test and control groups followed by regression analyses—showed that the new product would likely cannibalize other more profitable items.”
The combination of rigorous testable hypotheses followed by methodologically sophisticated experimentation to support business learning sounds suspiciously like “The Science of Business Manifesto” articulated several years ago.
In that blog, I noted “A key first step ... is transitioning from the hypotheses to testable and measurable constructs of the form “the more of A we do, the more of B that will result ultimately leading to more C.” Once hypotheses are operationalized, performance indicators can be formulated and mapped to existing data. Also central are the designs used to ensure that relationships between A, B and C are causal: A causes B which in turn causes C. Randomized experiments are the platinum standard for design, but other methods, such as natural time series with a comparison group, can be effective where randomization isn't feasible. Critical “deliverables” are both well-specified hypotheses and rigorous designs for testing.”
Perhaps a more eloquent science of business argument is made by Micha'l Bikard and Charles E. Eesley in their two-part series Exploring the Economic Experiment (I): Hypothesis Development, and Exploring the Economic Experiment (II): Hypothesis Testing.
Rather than as gut-driven heroes, B&M see entrepreneurs as more plodding scientists, proceeding from hypothesis to hypothesis, experiment to experiment. “the role of the entrepreneur mirrors the role of the scientist, who is developing hypotheses about the world....Like the scientist, the entrepreneur comes up with original ideas and attempts to prove their value through experimentation: only in this case, the entrepreneurial venture is the experiment.”
In fact, 'The idea of constantly pulling information from the market is not new, but can perhaps best be described as part of a more comprehensive and systematic conception of the entrepreneurial process as a type of hypothesis generation and testing, rather than viewing it as advice for product development only. The image of entrepreneurial strategy as a series of explicitly defined hypotheses to be tested and modified can be a powerful tool for the practice of this “economic experiment.” '
I find thinking about business in familiar scientific terms quite appealing. Translating entrepreneurship and strategy to theory, hypotheses and testing resonates well with a data-driven mindset, and provides a nice foundation for dialog between business and analytics.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access