My recent two-part IM series was generally well received, and I've recently engaged in a number of productive discussions with customers and prospects on the topic. Those who've adopted the balanced scorecard approach to strategy development/deployment seem particularly enthused, often mentioning theories, hypotheses and cause and effect linkages when discussing their businesses. And OpenBI's had success using the SOB method for requirements formulation with our BI and analytics customers.
A critical consideration surrounding the causes and effects of business strategy is the design that envelops a new intervention. The platinum standard is the experiment, in which the strategic interventions are applied randomly to some prospects/customers (treatment) but not to others (control). Randomized assignment to treatment/control assures within probability limits that the groups are “equal” out of the gate on factors other than the intervention. Measured differences between the groups can then be reasonably assumed to be the result of the intervention – i.e. that treatment A caused outcome B. Capital One's online experiments promoting new credit card packages illustrate the randomized design well, demonstrating which specific offers are responsible for customer growth.
Effective designs aren't limited to randomized experiments, however. In many cases, experiments are impractical, so alternatives to randomization are deployed. An illustration pertinent for BI is multiple time series, where treatment and control groups are “natural” – not randomly assigned. Multiple over-time measurements taken on each group both before and after an intervention are compared to calibrate the impact of treatment.
Designs relevant for BI can be compared on a number of dimensions, not the least of which is internal and external validity. In their text "Experimental and Quasi Experimental Designs," authors William Shadis, Thomas Cook and Donald Campbell offer the following definitions. “We use the term internal validity to refer to inferences about whether observed covariation between A and B reflects a causal relationship between A and B ... the researcher must show that A preceded B in time, that A covaries with B and that no other explanations for the relationship are plausible.”
“External validity concerns inferences about the extent to which a causal relationship holds over variations in persons, settings, treatments and outcomes.” In non-academic speak, internal validity addresses whether the intervention caused any noted measurement differences between treatment and control, while external validity addresses how findings can generalize to other situations.
You run across considerations of internal and external validity all the time. A recent column in Newsweek challenges the conventional wisdom of why the U.S. population is becoming increasingly obese. The generally-accepted explanation involves some combination of super-sized fast food diets, a growing electronic culture that crowds out physical activity, and a drive-everywhere suburban life style that encourages sloth. The linkage is that these activities in tandem (A) have caused the increase in population obesity (B).
But a researcher from Alabama has found evidence that animal populations have gained weight appreciably over the years as well, casting doubt on the accepted theory. “In a paper to be published in Proceedings of the Royal Society B (for biology), they report that in 23 of the 24 – eight species, 20,000-plus animals – the percentage of obese individuals has risen since the 1940s (or since the oldest records they found). The odds of that happening by chance are 8 million to 1. And since neither feral rats nor lab chimps nor any of the others have cut back on phys ed or patronized vending machines more, says Allison, we need to look for explanations beyond ...”
The Newsweek columnist goes on to propose new hypotheses that perhaps the cause is something common to both humans and animals over the last 50 years, such as an increase in sleep debt, which in turn stimulates appetite. Regardless of the results of new research, the internal validity of the conventional explanation is challenged by these findings, opening the door for new and competing hypotheses.
The weekly WSJ Research Report routinely addresses issues of validity in its brief articles on health care research. One study by Swedish researchers that evaluated 11,246 men and 2,858 women annually for 15 years, including an equal number of years before and after they retired, concludes that retirement reduces mental and physical fatigue and improves mood. In an important nod to external validity, the Journal caveats: “the average retirement age was 54.8 years, so the findings may not apply to people who leave the work force later.”
In another study cited from the New England Journal of Medicine, a meta-analysis of 19 U.S. studies involving 1.5 million participants age 19 to 84 concluded that those with a high body-mass index (BMI), especially before the age of 50, have a greater risk of dying from heart disease, cancer and other causes. All fine and good cautioned the Journal, except that: “The study looked at white people in affluent countries, and may not apply to other populations.”
BI analysts are well advised to similarly challenge their organization's espoused strategic theories, looking for alternative explanations to A causes B (internal validity) and for limits to how far A causes B generalizes to other situations (external validity).