I love watching football on those weekends in the fall when family demands are minimal. Sitting in front of the TV last Saturday and Sunday, Ihad the opportunity to sort through the accumulation of business journals I subscribe to, including WSJ, BusinessWeek, The Economist,Harvard Business Review, and Forbes. Alas, I'd fallen behind in the reading and was under pressure from the family to clean my periodicals house. What better way to catch up than by watching two college games on Saturday and two pro games on Sunday, all the while perusing wisdom from leading business pundits?


I'm not sure if BI is getting more press now than ever, but over the two days I came across eight to 10 articles that were quite pertinent for our audience. Early in Saturday's first game I decided to write October's column on BI in the business press. By the end ofsecond game, I had eliminated several articles as content, and by the end of Sunday's pro games decided to further limit my focus to experimentation in business.


Super Crunchers


Regular readers of the OpenBI Forum are probably nauseatingly familiar with my insistence on quality designs for establishing the validity of BI analyses (http://www.dmreview.com/news/1081924-1.html). Of course the platinum standard for design is a randomized experiment in which “subjects” are assigned by chance to treatment and control groups, thereby assuring that differences noted are due to the groups, not confounding factors. This argument was taken to another level of sophistication by Yale professor Ian Ayres in his latest book, Super Crunchers, which I reviewed in September. Ayres argues with a wealth of great examples that super crunching, the combination of predictive modeling and randomized testing, can provide significant lift for programs in government, education, health care and business.


Ayres shows his obsession with experiments again, this time with Yale colleague Barry Nalebuff, in a Forbes Magazine article of September, 3, 2007 entitled, appropriately enough, "Experiment." In this column, Ayres and Nalebuff argue that Wal-Mart, a noted BI juggernaut, could be even more successful with knowledge gleaned from randomized experiments than it is currently with historical observations. If, for example, Wal-Mart conducted tests of different retail space allocation ideas across randomly-assigned stores, the results would be more valid - and perhaps sustainable - than those from historical observation alone. Another illustration where experimentation could add to performance management is an insurance company's evaluation of its underwriting policies. The company will obviously use historical member and claims information for measurement, but that information alone cannot tell anything about prospects that were not accepted, only ones that were. Small randomized tests of those who would have been rejected, on the other hand, might reveal profitability foregone. Progressive Insurance, for example,found that middle-aged, college-educated motorcycle riders were excellent, and profitable, risks. The Ian Ayres mantra: randomized experiments for business are important adjuncts to expert opinion for the formulation of business strategy. And randomized testing, especially in the Internet age, may be much less intrusive - and more accessible - than often thought.


Tom Davenport, author of Competing on Analytics, agrees with Ayres and takes the thinking one step further. In a Wall Street Journal interview, Davenport responded to a question on the foundation for successful deployment of business analytics: “I'd say the number one factor is really a leadership dimension. It's how committed are a company's senior executives to fact-based and analytical decision-making and the whole idea of experimentation as a way to learn rather than doing it out of gut feel or intuition.”




I'm a big fan of Amazon and its CEO, Jeff Bezos. Since the advent of Amazon Prime, which essentially covers an account's unlimited annual shipping costs for $75, I buy my books, CDs, DVDs, camera equipment, holiday gifts - just about anything I can - from Amazon. Amazon's collaborative filtering recommendation engine has probably sold me a dozen books and DVDs over the last two years alone. I continue to be amazed that I can order a book at 4:30 p.m. and have it delivered by 10:30 a.m. the next day.


Even as Time Magazine's Person of the Year for 1999, when it seemed all the market experts were trumpeting a new economic order fueled by the Internet that was different this time,Bezos had his feet firmly planted on the ground, realizing his work to build a successful company was just beginning. Amazon is committed to germinating its strategy, patiently letting the model play out in the marketplace. And Bezos is not bullied by Wall Street experts on how he should run his company. The decision to launch Amazon Prime, decried as a money loser by analysts, has instead proven quite successful. What Amazon might be subsidizing in shipping costs it is more than recouping from the resulting deeper customer relationships. The market has acknowledged Amazon's success as well. The Amazon stock price is up fourfold in the past five years, doubling in the last year alone.


Little wonder then, that Bezos is the subject of a Harvard Business Review interview: "The Institutional Yes," October, 2007. For anyone interested in the strategic thinking of an e-commerce leader, this article is a great place to start. Bezos speaks of Amazon's obsession with customers as driving from a meta strategy to be “stubborn on the vision but flexible on the details.” Arguing that errors of omission are more grievous than those of commission, Bezos offers as antidote a corporate commitment to the experimental method. Indeed, for Amazon, randomized experiments are a fuel for strategy, providing answers to many questions that in other environments are relegated to flawed “experts.” Frequent Amazon Web site surfers can be certain they are participating in many randomized tests to optimize their experience - and the resulting business benefit.


As Bezos notes: “What you really want to do companywide is maximize the number of experiments you can do per given unit of time. ... Somebody wears the black hat and makes the case for why not to do it, and somebody else puts on the white hat and says why it's actually a good thing to do. But since the outcomes of all these things are uncertain, if you can figure out how to conduct an experiment, you can make more bets. ... We have a group called Web Lab that is constantly experimenting with the user interface on the website, getting statistical data from real usage patterns about which interfaces work best. This is a huge laboratory for us. ... You should be making guesses and then finding out at lowest cost whether or not they are needle movers.” Hopefully, a commitment to the testing method as a beacon of strategy for a company the stature of Amazon will provide assurance for others looking to implement experimental intelligence initiatives.


I Wish I Could Conduct an Experiment


I knew I had come to the end of my reading line when I encountered the BusinessWeek, October 22, 2007 article "I Can Get Your Kid Into a Ivy." With one son a college freshmanand another a junior in high school and already involved in college search, the entireadmissions adventure is front and center with me. If the article's authors intended the piece to be provocative or even inflammatory, they succeeded.


The admissions consultant who is the subject of the story is portrayed as having enough arrogance and hubris to make Salesforce.com CEO Marc Benioff blush. Claiming success rates in excess of 95 percent gaining client acceptance into first choice colleges, the consultant charges fees averaging almost $20,000 and as much as $40,000 for her services.“I'd be an idiot to charge half of what I can. Parents can always hire a lesser person.”


The consultant certainly has credentials for her role: she served for several years on the staff at Dartmouth, so is very familiar with the supply side of college admissions. Indeed, a book she published on the Dartmouth selection process was a springboard to her current business. Every indicator suggests she's uber competent in her role as advisor, able to best position and groom her candidates for their college choices. The parents who pay the steep fees are generally impressed with her work, as are the applicants themselves, though they often don't feel she's indispensable. College admissions officers, to little surprise, are not advocates. “I believe that most of the funds expended on independent counselors are simply wasted.” said Jeffrey Brenzel, dean of admissions at Yale.


Setting aside the question of whether the extreme fuss over elite colleges by families is productive - is an economics degree from Brown worth more than an economics degree from Wisconsin? - and setting aside concerns of additional advantage for the well-heeled, the BI analyst in me asks whether an admissions consultant makes a difference. Is the juice worth the squeeze?


Unfortunately for admissions consulting consumers, but perhaps fortunately for consultants, it would seem very difficult to divine a practical random experiment to evaluate the performance of counselors. I'm not sure affluent parents of highly driven high school seniors would want their kids randomly assigned to either admission consulting groups or no consulting controls. There's also the issue of “first choice.” If Moron Miller's first choice is Harvard, but the counselor convinces him that Lowly State is his only prospect, does that still constitute a first choice? A more likely design might be a quasi-experiment that, while not randomized, can still provide some valid evidence. Sounds like a job for Freakonomics author, Steve Levitt.


In the interim, let me propose two tongue-in-cheek impractical randomized experiments to test the performance of college admissions consultants. The first would involve all college-bound, high school senior pairs of identical twins with A- or higher GPAs and the equivalent of 28 or more on the ACT. One twin would be assigned randomly to an admissions consultant, the other would serve as a control with no paid admissions support. Come April, the admission consultants' success records would be compared among each other and the control group. Statistically significant higher success rates getting clients into superior, first-choice schools in contrast to the control would beindications the consultants are worth their exorbitant fees.


To evaluate just a few consultants, maybe we could solicit involvement from several of New England's prominent boarding high schools. Since practically all seniors at those schools are headed to college, many to elite private schools,we would randomly assign each student to a either a consultant or the control group that gets no paid help, comparing performance in April. Perhaps those advisors who don't measure up would refund their fees - or better, donate them to an educational charity.




  1. Ian Ayres. Super Crunchers – Why Thinking-By-Numbers is the New Way to Be Smart. Bantam Books. 2007.
  2. Ian Ayres and Barry Nalebuff. "Experiment." Forbes. September 3, 2007.
  3. Thomas H. Davenport and Jeanne G. Harris. Competing on Analytics – The New Science of Winning. Harvard Business School Press. 2007.
  4. Christopher Lawton. "Understanding What You Know: How Business Intelligence Has Come of Age." WSJ, October 23, 2007.
  5. Julia Kirby and Thomas A. Stewart. "The Institutional Yes."  An Interview with Jeff Bezos of Amazon. Harvard Business Review. October, 2007.
  6. Susan Berfield and Anne Tergeson. "I Can Get Your Kid Into an Ivy." BusinessWeek. October 22, 2007.
  7. Steven D. Levitt and Stephen J. Dubner. Freakonomics – A Rogue Economist Explores the Hidden Side of Everything. HarperCollins. 2005.





Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access