Biases in BI
In the absence of randomization which would reasonably assure that confounding factors are equal between treatment and control, quasi-experimental designs consider the sources of confounding bias specifically. Among the key potential confounders are:
History causes other than the treatment intervene between pre and post measurement.
Maturation a pre-post difference might be imputed to a study factor when in fact it simply represents the maturation of the subject.
Mortality subjects drop out of the investigation between pre and post, invalidating a comparison.
Selection the treatment and control groups differ systematically out of the gate on factors other than treatment that could explain differences.
Unintended treatments e.g. the Hawthorne effect, wherein subjects respond to the very act of measurement, independent of the intervention.
Regression to the mean Strong positive measures at time t1 might regress to the overall mean and decline at t2, independent of the intervention. Negative measures at t1, on the other hand, might improve at t2, the result of regression as well. Case in point: The new coach of last year's 0-16 Detroit Lions NFL team should reap the benefit of that futility next season. The Lions will undoubtedly win a few games in 2009, even if they're still awful.
Interrupted Time Series Quasi-Experimental Designs
The baseline quasi-experimental designs for business are the single group pretest-posttest and the two-group pretest-posttest with both treatment and control. As noted in my blog on Nonrandomized Experiments
The interrupted time series is a class of designs that provides a higher level of bias or confounder protection for BI by including multiple pre and post treatment measurements. And these designs are natural for business, where measurements are routinely made over time. With the simple interrupted time series, the analyst compares the multiple pretest measurements to the posttests to determine if an intervention effect may be confused with a trend. Without a control group, however, the analyst is hard pressed to refute the challenge of a potential history confounder. A BI example is the introduction of a CRM training program for employees, where the measurement could be customer attrition or lifetime value.
The two-group pretest-posttest with an untreated control group but multiple pre and post measurements also provides additional protection for BI. This design guards against history or maturation biases, but leaves the analysis exposed to the confounding of selection to treatment and control. Though not a panacea, this design is quite strong and should be a staple in every BI designer's tool chest.
The repeated-treatment designs add validity-enhancing twists to the two-group pretest-posttest with multiple pre and post measures. Instead of a single treatment preceded and followed by measurements, this design cycles through a series of treatment/removal of treatment iterations, with the before and after measures. An increase after treatment, followed by a decline with treatment removal, then an increase again post treatment restoration, provides evidence of an intervention effect. E-tailers might consider this design for more precise testing of new promotions.
Interrupted time series designs with a control group are often analyzed with techniques that attempt to statistically adjust measurements for the contaminating effects of known confounding variables. Procedures such as matching, analysis of covariance, and propensity models are used to help purify results, allowing these designs to approach the validity of randomized experiments. At the same time, the data can simply yet productively be analyzed visually using the powerful multidimensional statistical graphics capabilities available in packages such as R. Perhaps the BI analyst can use propensity models to satisfy statistical purists and statistical graphics for business consumers.