OpenBI Forum: Experiments in BI
Information Management Online, October 25, 2007
I love watching football on those weekends in the fall when family demands are minimal. Sitting in front of the TV last Saturday and Sunday, I had the opportunity to sort through the accumulation of business journals I subscribe to, including WSJ, BusinessWeek, The Economist, Harvard Business Review, and Forbes. Alas, I'd fallen behind in the reading and was under pressure from the family to clean my periodicals house. What better way to catch up than by watching two college games on Saturday and two pro games on Sunday, all the while perusing wisdom from leading business pundits?
I'm not sure if BI is getting more press now than ever, but over the two days I came across eight to 10 articles that were quite pertinent for our audience. Early in Saturday's first game I decided to write October's column on BI in the business press. By the end of second game, I had eliminated several articles as content, and by the end of Sunday's pro games decided to further limit my focus to experimentation in business.
Advertisement
Super Crunchers
Regular readers of the OpenBI Forum are probably nauseatingly familiar with my insistence on quality designs for establishing the validity of BI analyses (http://www.dmreview.com/news/1081924-1.html). Of course the platinum standard for design is a randomized experiment in which “subjects” are assigned by chance to treatment and control groups, thereby assuring that differences noted are due to the groups, not confounding factors. This argument was taken to another level of sophistication by Yale professor Ian Ayres in his latest book, Super Crunchers, which I reviewed in September. Ayres argues with a wealth of great examples that super crunching, the combination of predictive modeling and randomized testing, can provide significant lift for programs in government, education, health care and business.
Ayres shows his obsession with experiments again, this time with Yale colleague Barry Nalebuff, in a Forbes Magazine article of September, 3, 2007 entitled, appropriately enough, "Experiment." In this column, Ayres and Nalebuff argue that Wal-Mart, a noted BI juggernaut, could be even more successful with knowledge gleaned from randomized experiments than it is currently with historical observations. If, for example, Wal-Mart conducted tests of different retail space allocation ideas across randomly-assigned stores, the results would be more valid - and perhaps sustainable - than those from historical observation alone. Another illustration where experimentation could add to performance management is an insurance company's evaluation of its underwriting policies. The company will obviously use historical member and claims information for measurement, but that information alone cannot tell anything about prospects that were not accepted, only ones that were. Small randomized tests of those who would have been rejected, on the other hand, might reveal profitability foregone. Progressive Insurance, for example, found that middle-aged, college-educated motorcycle riders were excellent, and profitable, risks. The Ian Ayres mantra: randomized experiments for business are important adjuncts to expert opinion for the formulation of business strategy. And randomized testing, especially in the Internet age, may be much less intrusive - and more accessible - than often thought.
Tom Davenport, author of Competing on Analytics, agrees with Ayres and takes the thinking one step further. In a Wall Street Journal interview, Davenport responded to a question on the foundation for successful deployment of business analytics: “I'd say the number one factor is really a leadership dimension. It's how committed are a company's senior executives to fact-based and analytical decision-making and the whole idea of experimentation as a way to learn rather than doing it out of gut feel or intuition.”
Amazon
I'm a big fan of Amazon and its CEO, Jeff Bezos. Since the advent of Amazon Prime, which essentially covers an account's unlimited annual shipping costs for $75, I buy my books, CDs, DVDs, camera equipment, holiday gifts - just about anything I can - from Amazon. Amazon's collaborative filtering recommendation engine has probably sold me a dozen books and DVDs over the last two years alone. I continue to be amazed that I can order a book at 4:30 p.m. and have it delivered by 10:30 a.m. the next day.
Page 1 of 3.
