Analytical Designs for BI – Part 2

I was quite pleased as I started to write this column. I'd just gone out to the Information Management site and seen from the comments that no less than 3 people had read Analytical Designs for BI – Part 1. I'm pretty sure from tenor of the notes that those 3 will read this column as well. Progress.

Part 1 discussed randomized experiments for business performance measurement. Classic experiments, the platinum standard for BI investigations, are, unfortunately, not always practical in business settings. The less intrusive lottery designs, which deploy but mask randomization, may be suitable for an additional class of business evaluations. “Plan B”, when randomized experiments are out of consideration, involves quasi-experimental studies that attempt to control potential confounding variables by clever design techniques and statistical adjustment.

When designing quasi-experiments (or randomized experiments for that matter), BI analysts should be guided by considerations of internal validity of their investigations. Internal validity obsesses on threats to the proposed cause-effect relationship between experimental factors and the dependent measures. As G. David Garson, cited in Part 1, notes: “When there is lack of internal validity, variables other than the independent(s) being studied may be responsible for part or all of the observed effect on the dependent variable(s).

Biases in BI

In the absence of randomization which would reasonably assure that confounding factors are equal between treatment and control, quasi-experimental designs consider the sources of confounding bias specifically. Among the key potential “confounders” are:
History – causes other than the treatment intervene between pre and post measurement.
Maturation – a pre-post difference might be imputed to a study factor when in fact it simply represents the maturation of the subject.
Mortality – subjects drop out of the investigation between pre and post, invalidating a comparison.
Selection – the treatment and control groups differ systematically out of the gate on factors other than treatment that could explain differences.
Unintended treatments – e.g. the Hawthorne effect, wherein subjects respond to the very act of measurement, independent of the intervention.
Regression to the mean – Strong positive measures at time t1 might “regress” to the overall mean and decline at t2, independent of the intervention. Negative measures at t1, on the other hand, might improve at t2, the result of regression as well. Case in point: The new coach of last year's 0-16 Detroit Lions NFL team should reap the benefit of that futility next season. The Lions will undoubtedly win a few games in 2009, even if they're still awful.

Interrupted Time Series Quasi-Experimental Designs

The baseline quasi-experimental designs for business are the single group pretest-posttest and the two-group pretest-posttest with both treatment and control. As noted in my blog on Nonrandomized Experiments, these are generally inferior BI designs that should be deployed only when stronger candidates are impractical.

The interrupted time series is a class of designs that provides a higher level of bias or confounder protection for BI by including multiple pre and post treatment measurements. And these designs are natural for business, where measurements are routinely made over time. With the simple interrupted time series, the analyst compares the multiple pretest measurements to the posttests to determine if an intervention effect may be confused with a trend. Without a control group, however, the analyst is hard pressed to refute the challenge of a potential history confounder. A BI example is the introduction of a CRM training program for employees, where the measurement could be customer attrition or lifetime value.

The two-group pretest-posttest with an untreated control group but multiple pre and post measurements also provides additional protection for BI. This design guards against history or maturation biases, but leaves the analysis exposed to the confounding of selection to treatment and control. Though not a panacea, this design is quite strong and should be a staple in every BI designer's tool chest.

The repeated-treatment designs add validity-enhancing twists to the two-group pretest-posttest with multiple pre and post measures. Instead of a single treatment preceded and followed by measurements, this design cycles through a series of treatment/removal of treatment iterations, with the before and after measures. An increase after treatment, followed by a decline with treatment removal, then an increase again post treatment restoration, provides evidence of an intervention effect. E-tailers might consider this design for more precise testing of new promotions.

Interrupted time series designs with a control group are often analyzed with techniques that attempt to statistically adjust measurements for the contaminating effects of known confounding variables. Procedures such as matching, analysis of covariance, and propensity models are used to help “purify” results, allowing these designs to approach the validity of randomized experiments. At the same time, the data can simply yet productively be “analyzed” visually using the powerful multidimensional statistical graphics capabilities available in packages such as R. Perhaps the BI analyst can use propensity models to satisfy statistical purists and statistical graphics for business consumers.