Validity, Design and BI, Part 3
The May OpenBI Forum is the third of a series. For the second time this year, I've changed the topic of a column at the 11th hour based on something I've recently read. This time, that something is a business book, The Halo Effect - and the Eight Other Business Delusions That Deceive Managers, by Phil Rosenzweig. The Halo Effect, an easy three-hour read, is destined to be one of the best business books of 2007 - or any year for that matter. And since much of the book's content challenges the validity and interpretation of popular business research, its concerns are quite similar to those of the OpenBI Forum's two latest columns on validity, design and BI.
Readers who derive their corporate opinions from Forbes, Fortune and Business Week, or who lionize the popular management tomes of the last 25 years like In Search of Excellence, Built to Last, What Really Works and Good to Great, should be a bit circumspect when they take on The Halo Effect. For without inflammation but with surgical precision, Rosenzweig debunks the feel-good findings and interpretations of this research, leaving readers with the near-certain conclusion the books are little more than nice stories of how the business world should work. Alas, as The Halo Effect demonstrates, real business isn't as tidy or conforming as we'd like.
The Halo Effect
In psychology, the halo effect is an error in interpretation in which an evaluation of people or organizations on one dimension is generalized to other dimensions without substantiation. The halo effect was coined by psychologist Edward Thorndike during World War I to explain the phenomena where soldiers who ranked highly on leadership and intelligence, say, were assumed to be superior in other traits, while those deemed lacking in leadership were seen as inferior across the board. Over time, the halo effect has come to represent a tendency to assume specific traits based on a general impression - to blend features together, to make attributions from what is known to what is vague.
Rosenzweig argues convincingly that the halo effect is in play when companies whose financial performance is superior are presumed to have a laser-like strategic focus, to execute with aplomb, to have enabling leadership, to possess a cohesive culture and to be locked in on customer needs. Companies struggling financially, on the other hand, are assumed to have strayed from core, to be arrogant, to have lost focus, to be disconnected from customers. The big problem for the type of analyses noted in The Halo Effect is that indicators of company performance such as leadership, strategic focus, culture and customer orientation are not measured objectively and independently of financial performance, but rather assume the "halo" of that financial performance - either positive or negative.
Rosenzweig's discussion of the up and down fortunes of Cisco and the Swedish-Swiss industrial company ABB is as entertaining as it is illuminating. Citing popular business press, Rosenzweig illustrates the halo effect with all its splendor. The CEO of ABB, characterized as charismatic and visionary while ABB's performance was in orbit, is seen as arrogant and imperial on the decline. ABB's decision to expand into new areas like financial services was framed as a bold growth move in good times, but criticized as impulsive and foolish on the downturn. Cisco, lauded for its culture, customer focus, acquisition strategy and leadership in the high-flying late 90s, was denigrated as an out-of-control and overhyped underachiever in 2002. The juxtaposition of before and after quotes on the "causes" of the fortunes of ABB and Cisco - the halo effect in practice - is almost comical.
Though a psychological process that causes people to assess in certain ways, the halo effect needn't be an inevitable consequence of our desire to evaluate, our need to examine cause and effect. Indeed, the biggest problem with the assessments and studies noted in The Halo Effect is one of design and method. The dimensions of company performance in these cases - strategy, leadership, culture, etc. - are measured not by independent, objective evidence, but by individuals who are subject to a biasing halo interpretation. The inferior designs of these anecdotes, case studies and cross-sectional investigations almost guarantee that dimensional assessments will be reflective of overall performance, thus assuring a halo effect.
The halo effect is not the only validity-threatening flaw discussed in the book. Nine delusions are noted in all, among them "correlation and causality," "lasting success," "connecting the winning dots" and "the wrong end of the stick." The problem of correlation and causality is as old as science and has to do with interpreting an association between two variables. Does a strong relationship between employee satisfaction and company performance imply that satisfied employees enhance company performance? Or that good company performance makes for satisfied employees? Or both? Or neither? The directions of many such relationships between company performance and dimensions of performance noted in The Halo Effect are certainly open to debate, especially those from anecdotes or cross sectional investigations that measure at one point in time only.
The delusion of lasting success embodies the statistical artifact of regression to the mean, in which above-average performance tends not to persist, while below-average results tend to improve over time. Nowhere is this more evident than in stock prices, often the top performance measure cited in the various analyses and studies of The Halo Effect. Rosenzweig notes that over half of Built to Last visionary companies failed to match the S&P 500 over the five year period post-study, after outperforming the index by a factor of 15 for 60 years. This type of finding is commonplace in the markets. Little wonder prudent investment managers urge customers not to chase outstanding stock performance.
Connecting the winning dots, an explication of the highly acclaimed In Search of Excellence, is a discussion of the perils of case control research, in which a sample pre-selected on a set of attributes - "successful" companies in this case - comprises the entire study population. A more comprehensive case control design would match successful companies against unsuccessful ones, searching for both similarities within and differences between. The characteristics determined to differentiate "excellent" companies in this study include "staying close to the customer," "stick to the knitting," "a bias for action," etc.
The wrong end of the stick relates to interpretations presented in Good to Great, a "complete" case control investigation that contrasts two preselected groups: "great" companies and "good" companies. The metaphor of comparison is Isaiah Berlin's work of The Hedgehog and the Fox, in which: "the fox knows many things, but the hedgehog knows one big thing." In the business world, hedgehogs are seen as plodding, but methodical with a single idea, a razor-sharp vision, while foxes are fleet and cunning but diffuse in purpose, attending to many activities, with many ideas. One of the major findings of this study is that great companies are disproportionately hedgehogs with a purity of focus, while just good companies are over-represented by diffuse foxes. Interestingly, Berlin sides more with foxes than hedgehogs in his discussions on the history of ideas.
The Evidence Hierarchy
The halo effect, correlation and causality, lasting success, connecting the winning dots, the wrong end of the stick - and the other delusions noted by Rosenzweig as well - are all illustrations of flawed interpretations resulting from the inferior designs and measurement of analyses and research on business performance. Indeed, in almost every case noted, the authors far overstep the interpretation boundaries of their study designs and measures, confidently positioning one of but many plausible explanations for their findings.
The fields of epidemiology and biomedical science offer a ranking or hierarchy of study designs that can bring much to bear to both business intelligence and the research illustrated in The Halo Effect. Figure 1, an "Evidence Hierarchy" from the Web site of the School of Health and Related Research at the University of Sheffield in the UK, ranks different designs by the severity of potential threats to the validity of research findings, with 8) Anecdotal, the least desirable "design," and 1) Meta-Analyses, the most desirable. Each design should be evaluated by criteria we've used in prior columns, such as selection, history, and regression to the mean. In Part 2 of the OpenBI Forum series on Design and Validity for BI, we noted the primacy of randomized experiments and panel studies for business intelligence - designs 2 and 3 in the Evidence Hierarchy. Alas, the designs examined in "The Halo Effect" fall exclusively in the inferior categories 4 to 7. The business performance analyses noted in "The Halo Effect" range from the expert opinion of the business press to business school case studies to the cross sectional and case control studies of popular management gurus. Measurement is often not much better than the weak designs, relying on the halo of expert opinion to assure that performance measures and dimensions align. "The Halo Effect" correctly takes exception to the bravado promoting conclusive findings for business studies from such weak methodology.
Figure 1: Evidence Hierarchy
The lessons distilled from The Halo Effect are as important for BI as they are for business research. Both researchers and BI analysts must obsess with the quality of designs of their inquiries to ensure the validity of findings. While the gold standard of randomized experiments (2 in the Evidence Hierarchy) may be impractical for many business situations, such experiments may, in the Internet era, be more accessible than generally thought. Cohort or panel studies (3 in the Evidence Hierarchy) involving multiple groups over time are often an acceptable alternative to experiments, able to withstand challenges of selection, history, and regression to the mean. Those whose designs are consigned to case-control studies or less (4 to 7 in the Evidence Hierarchy) must be especially wary of overstating findings, mindful that credible competing explanations could substantially undermine their work. Unfortunately, the analyses described in The Halo Effect may well be tarnished by precisely such mistakes.
1. Phil Rosenzweig. The Halo Effect - and the Eight Other Business Delusions That Deceive Managers. Free Press. 2007.
2. Thomas J. Peters and Robert H. Waterman Jr. In Search of Excellence: Lessons from America's Best-Run Companies. Warner Books. 1982.
3. James C. Collins and Jerry I. Porras. Built to Last: Successful Habits of Visionary Companies. HarperBusiness. 1994.
4. Jim Collins. Good to Great: Why Some Companies Make the Leap ... and Others Don't. Random House Business Books. 2001.
5. William F. Joyce, Nitin Nohria and Bruce Robertson. What Really Works: The 4+2 Formula for Sustained Business Success. HarperBusiness. 2003.