I’ve subscribed to the Harvard Business Review for about five years. When the monthly magazine arrives in the mail, it often seems there are either several articles pertinent for business intelligence or none at all. The February 2009 edition was one of the former.

The article: Why Good Leaders Make Bad Decisions, cites neuroscience research to observe that leaders often make decisions through the unconscious processes of pattern recognition and emotional tagging. Pattern recognition use assumptions from prior experiences to categorize a current decision situation, often suggesting solutions similar to those that worked in the past. Emotional tagging is about the emotionally-committed preferences of the decision-maker, which of course can have substantial impact on the action taken. These processes, which in ways are similar to rules of thumb or heuristics, may produce effective decisions. They may also, however, be sources of systematic bias that can lead to faulty decisions.

The article cites examples from a book by one of the authors, Sydney Finkelstein, of Dartmouth’s Tuck School of Business, that conducted post-mortems of flawed business decisions. The articles’ authors suggest that businesses need to build safeguards into their management decision processes driven by red flag conditions to guard against such sources of bias.

Tom Davenport’s article: How to Design Smart Business Experiments, offers a scientific antidote to the “on a wing and a prayer” approach to decision-making. The culture of hypothesize/experiment/learn for operational decisions is closely aligned with the Evidence-Based Management philosophy espoused by Stanford professors Jeff Pfeffer and Bob Sutton. Indeed, the hypothesize/test foundation for business decisions is promoted in the Balanced Scorecard, Super Crunchers, and Enterprise Decision Management.

Davenport espouses a cycle for putting ideas to the test that includes:

  1. Create/Refine Hypotheses
  2. Design Experiment
  3. Execute Experiment
  4. Analyze Results
  5. Plan Rollout
  6. Rollout

Findings from all steps in the process are submitted to a Learning Library for posterity and, hopefully, for reuse.
Testing generally makes the most sense for smaller, operational decisions that are repeated often – the core of business transactions. At eBay, Amazon, and Google, randomized testing is the norm for website development. Sears has tested several formats for including its merchandise in Kmart stores, and vice-versa. Capital One has been in the testing vanguard since 1988, using experiments to design new offerings, moving to the top ranks of credit card companies by its “ability to turn a business into a scientific laboratory…subject to testing using thousands of experiments.” And Harrah’s Entertainment has given teeth to its hypothesize/test/learn culture by a mandate that “not using a control group” is rationale for termination.

Business Intelligence, which distinguishes between exploratory and confirmatory analytics, loves the focus given its efforts by an evidenced-based culture built on hypotheses and experimentation.  Evidence-based companies generally embrace BI early and significantly; return on investment (ROI) exercises for BI are strategic and substantive.

Steve Miller's blog can also be found at miller.openbi.com.