Related Links

Web Seminars

I was quite excited when my partner sent me the briefing documents from our newest customer, an eMarketer. I’m generally vigilant for opportunities to do analytics and predictive modeling, and so was heartened by the inclusion of two slide decks pertaining to propensity models and scoring. I guess I have a propensity for statistical analyses!

Simple Research Designs for Business Intelligence

Statistical models are generally developed in the context of research designs that allow results to be established with more or less confidence. The tighter the design, the more assurance the analyst can have of the findings. The gold standard for establishing the validity of an investigation is, of course, the randomized experiment. Randomization to treatment helps assure that observed differences in performance variables between experimental and control groups are due to the intervention and not to other uncontrolled factors (covariates), either observed or unobserved, that might be related to the performance measures and, subsequently, be sources of bias. With randomization, those bias-causing, uncontrolled factors should, on the average, be a wash between intervention and control groups.

At a minimum, business intelligence (BI) practitioners should understand the strengths and weaknesses of the designs they deploy to gather intelligence. Consider the six simple designs often used for BI outlined in Figure 1: where O represents observation or measurement, X is a treatment or intervention, and R stands for randomization. Design 1a, the one-shot case study, which offers no possibilities to learn from comparisons or overtime contrasts, is really not much of a design at all.  Yet this “design” is, unfortunately, quite pervasive in BI, underpinning much of predictive modeling, and a significant foundation for findings that impact business decision-making. The one group pretest-posttest 1b provides at least a pre-post comparison of the investigation units (customers, stores, etc.). The main problem with 1b is that differences in the pre and post measurements might be due to factors other than the intervention – and this design is hard pressed to refute alternative explanations.

Both pure experimental designs 2a and 2b should be standards by which BI aspires to gather intelligence. The power of randomization of units to either intervention or control groups, along with the benefits of pre and post measurements, make these simple designs well able to withstand threats to the validity of inquiries. And, in the Internet age, it’s often pretty straightforward to execute simple randomized experiments that can assure the quality of results.

For those cases in which randomization is impractical or inappropriate, quasi-experimental designs 3a and 3b, supplemented by statistical adjustments for bias, might be acceptable substitutes. Designs 3a and 3b introduce a next level of complexity to pre-experiments by adding a comparison or control group to the analysis. Indeed, quasi-experimental designs look much the same as their pure experimental cousins, except that they use natural groups instead of randomization to intervention/control. Without the benefits of randomization, selection and other biases can distort findings, misleading analysts to conclude there are differences between intervention and control, when in fact the groups are different (there are biases) out of the gate.

Figure 1

Propensity Models

There are many flavors of propensity models in the BI world today, each associated with one or more of the designs in Figure 1. Historically, marketing has equated propensity to predictive models that assess customer probability or likelihood of executing a critical event, such as purchase of goods and services. They speak of propensity (or inclination) for up sell and cross-sell. They trumpet lift, which is actually bang for the predictive buck, the hope being that a relatively small and predictable group of prospects makes the lions’ share of purchases. With significant lift come cheaper modeling, superior predictive accuracy and noticeable marketing ROI.