Super Crunchers - Lessons for BI Learned and Confirmed

OpenBI Forum

Information Management Online, September 28, 2007

Steve Miller

I almost couldn't get myself to purchase a book called Super Crunchers. The thinking was that such a title is more indicative of a Fox News Channel special investigation piece than a book on business intelligence (BI). Rupert Murdoch the new BI guru? It just doesn't seem right. Fortunately, you can't always judge a book by its title. Turns out there's a lot to like about Super Crunchers - and lots of lessons for BI, including several noted in previous OpenBI Forums. Turns out as well that the author eats his own dog food, with even the Foxian title the result of "super crunching."

If Competing on Analytics offers a solid foundation and conceptual framework for analytics in business and Smart (Enough) Systems provides painstaking detail on how to use analytics, decision rules and optimization techniques to automate routine decisions, Super Crunchers provides breadth of exposure to the current state of analytics in business, government, education and health care, offering incisive commentary on both developments and direction. Indeed, a dilemma for reviewers of Super Crunchers is deciding just what to exclude from consideration.

The Days of Wine and Numbers

A law professor and econometrician at Yale, Super Crunchers author Ian Ayres begins the discussion of predictive analytics with, appropriately enough, a modeling "application" developed by an economist that illustrates many of the issues surrounding super crunching. Orley Ashenfelter, Princeton professor and wine aficionado looking to optimize his selection of Bordeaux for purchase, developed a regression model to determine characteristics associated with vintage auction prices. His final equation reduced to three factors related to the growing season in France: winter rainfall, average temperature and harvest rainfall. The beauty - and the source of controversy - with this approach is that it allows Ashenfelter to predict the future quality of a vintage year before the wine is sold. And his predictions, published in a semiannual newsletter, have proven quite accurate - thus roiling the wine critics' community. One expert decried Ashenfelter's work as "rather like a movie critic who never goes to see the movie but tells you how good it is based on the actors and director." Fortunately, many who at first scoff at such unsophistication grudgingly relent to co-existence in time. The theme of predictive models versus resistant expert opinion or intuition is a recurring one in Super Crunchers.

The models of Orley Ashenfelter are now legend in college econometrics and social science statistics courses, and when subsequent examples in Super Crunchers included the usual competing-on-analytics suspects Netflix, Harrahs, Amazon and baseball's Oakland A's and Boston Red Sox, I began to fret that Super Crunchers was just a me-too analytics-in-business book. Not to worry. The supply of excellent examples seems almost endless.

The wine expert's analogy of Orley Ashenfelter to a movie critic who never watches films is eerily prophetic of the mission of start-up company Epagogix to provide analytic services to the film industry. Epagogix purports to predict the gross revenues of proposed motion pictures before filming has started, using scripts only and no indications of actors or directors. With a foundation of neural net data mining technology, models developed not only predict which films are not worth making, but also provide insight for tweaking to optimize financial intake. The models have done a pretty decent job back-testing the relationship of film characteristics with revenues. One consistent finding: film stars aren't worth their cost. The Epagogix CEO projects that major studios that would adopt the model's film-making discipline would net an additional billion dollars per year. Not surprisingly, film executives and staff aren't enthusiastic about the Epagogix story. "Nobody, nobody - not now, not ever - knows the least goddamn thing about what is or isn't going to work at the box office," huffed screenwriter William Goldman.

Random Analytics

Super Crunchers differentiates from most of the current literature on business analytics in its obsession with randomization to test alternative hypotheses/strategies/decisions. OpenBI Forum readers hear this ad nauseaum http://dmreview.com/article_sub.cfm?articleId=1081924, but random assignment to various treatment groups helps assure that groups are "equal" on factors other than the treatments, thus enabling valid interpretation of findings. Super Crunchers provides several good illustrations of the value of randomization in the financial services industry with CapOne and Credit Indemnity, but I was most intrigued by Offermatica, who's built a business on a "testing-driven approach to Web site design enables marketers to optimize their site's content and offers, to maximize their site's overall profitability." In one illustration, two alternative graphic designs were contrasted for ultimate consumer spend - and one was declared a clear revenue winner. With Offermatica, it's testing and analytics versus the opinions of Web usability experts, and the recommendations are often contradictory - and probably contentious. Super Crunchers wonders why randomization techniques are not even more pervasive in business. My guess is randomization and experimentation might in fact be more common than is generally acknowledged, especially for e-commerce companies. Such firms, though, may wish to keep their testing strategies confidential.

Do Government Programs Work?

Super Crunchers hits its stride with discussion of the effectiveness of government programs, an area of much academic attention over the years, starting with the work of the late Senator (and former professor) Daniel Patrick Moynihan 40 years ago. Indeed, it's not a stretch to say that super crunching for business owes a debt of gratitude to the mandates of Great Society programs to rigorously evaluate their efficacy. A new wave of experimental social scientists is paving the way for the next generation of policy initiatives. Esther Duflo, an economist with the Poverty Action lab at MIT, is co-instructor of the Putting Social Studies to the Test OpenCourseWare curricula reviewed by the OpenBI Forum in June http://dmreview.com/article_sub.cfm?articleId=1087121 .

Page 1 of 3.