Ive always been fascinated by findings in the academic and business worlds that arent immediately intuitive. As an undergraduate many years ago, one of my professors published a paper entitled The Strength of Weak Ties that has subsequently become a classic in social theory. In the article, Mark Granovetter argues convincingly that weak ties relationships more like acquaintances than strong friendships are important because they decrease the overlap in personal networks, and potentially enhance both the reach and cohesion of social relations.1 Weak ties thus facilitate the exchange of information across disparate networks of dense and strong relationships. In subsequent research, the author found corroboration for his theses by examining employment markets, in which candidates are often referred to new jobs through weak ties. I guess Im proof positive of this finding: over the past six months, Ive made successful referrals for two candidates whom I only know through email introductions by friends. So weak is good!
I initially found several of the tenets of modern portfolio theory (MPT) to be less than intuitive as well. The fact that higher performance portfolios are associated with increased risk is not surprising. But the ability to mitigate much of that risk by holding many assets whose returns are weakly correlated by diversifying didnt seem apparent to me at first. And actually strengthening a portfolio by adding lower-performing, less volatile and uncorrelated assets didnt quite get through the first time either. It wasnt until I understood the mathematics of why moderate returns with low volatility can often trump high returns with more variation that it started to make sense. So, diversity and independence are good, while volatility is bad.
The Wisdom of Crowds
Weak ties and diversified portfolios also play a central role in James Surowiekis excellent book, The Wisdom of Crowds. Surowieckis main thesis is that under certain controlled conditions, groups can make estimates and decisions that are, in many cases, superior to those of individuals even experts. Starting with an experiment by British scientist Francis Galton at a stock exhibition in London in 1906 in which 787 contestants had little success guessing the weight of an ox, but the average of all contestants was within one pound of the actual weight, Surowieki builds a compelling case for the wisdom of crowds.2
Crowd wisdom, however, is not a given in all circumstances, as riots and panics attest. Surowiecki discusses four conditions necessary to promote superior crowd decision-making. The first is diversity of opinion. Each individual in the group should have private information, even if its eccentric. That information held by individuals should be independent, so that opinions are not colored by others in the group. The group should be decentralized, allowing members to specialize and draw on local information. Finally, there needs to be some mechanism, such as polling, to aggregate the individual opinions into a collective decision.3
The factors promoting crowd wisdom demonstrate further the strength of weak ties and power of diversified portfolios. Crowd wisdom for decision-making is optimized when the collective is loosely tied, acting like individuals working independently, drawing on specialized knowledge of their networks with diversified opinions not unduly impacted by persuasion of the group. It is then that the power of statistical aggregation can often provide superior decisions and predictions to those of individuals.
Is there a role for the wisdom of crowds in business intelligence (BI)? Starting with experts versus analytics that Ian Ayres discusses in Super Crunchers, perhaps a third vantage point derived from the rigor of both analytics and individual assessment can be brought to bear on corporate decision-making.4 Indeed, that combination of individual decisions and statistics is precisely the nature of decision (prediction) markets, an intelligence technique gaining popularity with analytically-focused companies today.
The simplicity and efficiency of horse race gambling and sports bookmaking for aggregating the opinions of the masses provides a prototype for decision markets in which individuals bet on the outcomes of future events. One such example is the Iowa Electronic Markets (IEM), founded in 1988 to predict election results and run by the College of Business at the University of Iowa. The IEM allows participants to buy and sell futures contracts on candidates to either predict the winners of elections or the vote share per candidate. Since its inception, the IEM has proven remarkably accurate in predicting election results. A similar concept, the Hollywood Stock Market Exchange (HSX), allows the public to wager on film box office returns and Oscar winners. Again, the predictive results have been exemplary.
These same types of markets can be institutionalized and used for BI in the corporate world. In a recent Harvard Business Review Article on Googles Innovation Machine, authors Byla Iyer and Thomas Davenport discuss the companys commitment to fact-based management, noting an obsession with analytics and randomized experiments. The authors also detail Googles deployment of 300 prediction markets serving competitive intelligence from the wisdom of panels of employees. The markets provide estimates used by Google to forecast both demand for new products, such as Gmail, as well as the performance of competitive products like the Apple iPhone. The authors note the internal success of the prediction markets at Google, fundamentally enabled by the evidence-based mantra of management.5
As analytically sophisticated as Google is, it was not the first business to strategically deploy decision markets. Hewlett Packard has evolved applications in sales and order forecasting and now uses prediction markets in several business units. The HP methodology, behaviorally robust aggregation of information networks (BRAIN), is an adaptation of a true market and has proven quite successful.
Though the use of decision markets is growing and demonstrating a consistency of accurate predictions, the technique remains surprisingly underutilized as a strategic weapon for corporate intelligence. As Surowiecki laments, the most mystifying thing about decision markets is how little interest corporate America has shown in them. Corporate strategy is all about collecting information from many different sources, evaluating probabilities of potential outcomes, and making decisions in the face of an uncertain future. These are the tasks for which decision markets are tailor-made.6 Perhaps continued documentation of the successes of decision markets in the business world by academics and authors like James Surowiecki will hasten adoption of this promising intelligence technique.
Part 2 of the BI Ensemble series will discuss wisdom of crowds concepts applied to enhancing statistical and machine learning predictions.
- Mark S. Granovetter. The Strength of Weak Ties. The American Journal of Sociology, May 1973.
- Jack C. Francis and Roger Ibbotson. Investments A Global Perspective. Prentice Hall: 2002.
- James Surowiecki. The Wisdom of Crowds. Anchor Books: 2004.
- Ian Ayres. Super Crunchers. Bantam Books: 2007.
- Bala Iyer and Thomas H. Davenport. Reverse Engineering Googles Innovation Engine. Harvard Business Review, April 2008.