I often ask customers and BI colleagues if and how they know their BI initiatives are successful. Most can’t answer definitively, pointing to esoteric ROI calculations from finance or personal experiences within their companies. Others cite generic satisfaction surveys from analytics vendors or media like Information Management.

As I’ve noted in previous writings, BI/analytics surveys are fraught with problems of “representativeness”: does the sample adequately capture the population it purports to measure?  

Even if the answer to this question is yes, there are concerns about the validity of using surveys to measure both process and outcomes of an intervention like the use of BI/analytics. Questionnaires on the effectiveness of BI can easily by afflicted by a “halo effect” where respondent assessment of the business value of BI/analytics jives with other measures of satisfaction with the BI initiative. If their experiences are positive, then BI/analytics is performant; if they’re negative, then BI/analytics is not doing the job. The responses tend to be “haloed” to all positive or negative assessments.

A better methodology that probes for both process and outcomes is one that augments survey findings on practices with independent, objective assessments of results. A good example of this is articulated in a study I recently found: “Strength in Numbers: How Does Data-Driven Decision-making Affect Firm Performance?” by researchers from MIT and Penn. As the title suggests, the study attempts to test the hypothesis that a significant organizational commitment to evidence-based decision-making is associated with more successful firm productivity and financial performance.

For this cross-sectional study, business practice and information system measures were formulated from a survey administered by the researchers in conjunction with McKinsey and Company. The target population was senior human resource (HR) managers and chief information officers (CIO) from large publicly traded firms in 2008. The survey probed on business practices as well as the organization and usage of the information systems function.

The critical independent variable, data-driven decision-making (DDD), was constructed from three survey items having to do with using data for new products and services; the existence of data for decision-making across the organization; and the usage of data for business decision making across the organization. DDD was standardized to discriminate companies with a significant commitment to data-driven decision-making from those without.

For important hard measures such as physical assets and number of employees, as well as performance measures like sales and operating income, the researchers used the Compustat Industrial Annual file from 2005 to 2009. This information was supplemented by data from firm websites, the Orbis database, and www.answers.com.

With the independent practice and dependent performance variables in place, the authors turned to econometrics to estimate multifactor productivity equations using regression-related techniques. They fit a variety of models relating measures of organization output such as sales, profitability and market capitalization to firm inputs such as capital, labor, information technology capital, labor, other control variables and the DDD index. The “test” involved whether the effect of DDD would be statistically significant in the presence of the other inputs.

The results of the analyses are indeed gratifying for evidence-based management. The regression models suggest that firms with one standard deviation higher score on the DDD measure are, on average, about 4.6% more “productive” than their competitors. This productivity estimate persists even after controlling for IT and other inputs. Similar findings on the impact of DDD are also shown for the dependent measures of profitability and market value.

Though there’s a lot of “academese” in this study, I like its deployment of sophisticated methods and models to determine the impact of data-driven decision-making (DDD) on company performance. It’d be nice to see more utilization of methodology like this, linking surveys to objective performance measures in BI/analytics research, going forward.