Francis Galton was apparently on to something a hundred years ago when he discovered during a county fair contest to guess the weight of an ox that the average of all estimates was closer to the ox's true weight than the individual guesses of participants – and also closer than those of cattle experts.
In his important 2004 book, “The Wisdom of Crowds,” James Surowiecki, elaborates on this notion of crowd wisdom – and how lessons learned might be used to improve decision-making.
Since reading WC, I've become intrigued by crowd wisdom in my own work with predictive analytics. Ensembles often produce enhanced predictive performance by estimating many model variants, then combining them in some way. Bagging produces multiple models independently and averages the results. Boosting, on the other hand, fits a series of models in succession, each one in turn looking to predict results not well handled by those that preceded it.
It was only a matter of time before an enterprising company brought a product that revolves on the wisdom of crowds to the market. That vendor, Crowdcast, is positioning a social BI or collective intelligence platform that uses a betting game environment with virtual currencies to capture anonymous, unbiased individual opinions – “water cooler wisdom” – and produce tight forecasts that help companies manage their performance. Pertinent applications include competitor intelligence, project management, product development, risk management, portfolio management, and sales and marketing metric forecasting.
I stumbled across Crowdcast at the Business Analytics Summit in Chicago at the end of May, but was unable to take in the presentation by CEO Mat Fogarty. I sent an inquiry to the company and was quickly connected with Mat and Chief Scientist, Leslie Fine. We set up a conference call where Mat graciously agreed to do an interview with me for the IM audience. My questions and Mat's answers follow:
What’s the vision for Crowdcast – what will it take to get there?
“The vision is to tap team intelligence and convert it into the most reliable forecasts available. To do this, we need to continue to learn about the best conditions for generating predictions from a group of people. We have partnered with IARPA (the Intelligence Agencies Research arm) and academics from U.C. Berkeley and Wharton Business School on a 4-year research study to truly understand how to generate accurate predictions.
We need to integrate our solution into reporting and BI platforms to enable employees to predict metrics with the context of the data and analytics available. In this way, we make analytics more engaging and also create a single work flow for planning, reporting and forecasting. Lastly, we need to partner with innovative CIOs to fully realize the untapped intelligence of their employees. Crowdcast is at the forefront of a revolution to change the way information flows, and allies are required. Your people know … So ask them.”
In “The Wisdom of Crowds,” James Surowiecki articulates preconditions for wise crowds. These include independence, diversity and decentralization of opinion, as well as a means to aggregate individual judgments to a collective. Are these important to Crowdcast's business applications of collective wisdom? Is there a behavioral model behind Crowdcast's collective intelligence?
“These tenets are valid if we are optimizing for extracting wisdom from a diverse and heterogeneous crowd. Ideally, if information is uniformly dispersed, then the perfect cohort is as the book suggests.
However, our work is focused on helping corporations aggregate internal team knowledge. We've discovered over the years that the more pressing challenge is giving team members a safe and flat way to voice their knowledge and concerns. Our cohorts are usually the team members involved in the project, but also those tangential, providing a more independent opinion.
Our mechanism is designed to ensure not only to efficiently aggregate information, but to do so as simply as possible with as minimal an 'ask' on participation as possible. It asks meaningful questions in a very familiar vernacular, but is backed by a powerful aggregation algorithm that fully leverages all the information. We also gather commentary, and publish it along with the beliefs, providing a rich source of information for decision-makers.
The most important hurdle is ensuring that you are capturing the intuition of the team as frictionlessly as possible. To that end, we are moving towards a fully-integrated solution, where we solicit beliefs within business intelligence applications. Analysts, executives, and other team members logged into the BI system can contribute their insights in situ, and we then create a new data source alongside the traditional one.”
Does social BI compete with or complement traditional BI?
“BI is all about aggregating data into one place. Crowdcast is about aggregating the insights and opinions of the future into one place. The two make a perfect complement. For example, imagine an analytics dashboard with sales results and market share information per region. The analyst or manager looking at the dashboard now has the information on which to base a forecast – for example, Q1 sales were 110, Q2 were 130, what do you think Q3 will be? Clicking on the sales chart will bring up the question (in Crowdcast). The ability to capture insight at the time of the analysis is where collective intelligence will really shine.”
In traditional BI and Analytics, the wisdom of “experts” is often juxtaposed unfavorably to the cool analytics of predictive models. How would you characterize collective intelligence in the expert-analytics continuum? Is it some of both? Might collective intelligence be more suitable for strategic decisions while BI is better for operational?
“Data centric BI producing predictions (think predictive analytics) is often better for questions where you have a lot of historical data. For example calculating sales for established products. Collective intelligence is great for instances where there is a lot of human centered intuition involved, like sales of innovative products or ship dates of large projects. Also in times when the market is changing (like 2008-09) a human centered approach means that the metrics can more rapidly change given the situation.
We tend to only ask the employees a few questions – the maximum we recommend is around 40 at any one time. Because of this, it's better to focus on the more important decisions where the outcome is more valuable. In addition, employees are often more knowledgeable and interested in the bigger questions.”
Crowdcast was recently named to the prestigious MIT Technology Review TR50, the 50 most innovative companies for 2011. The TR50 is for organizations that “are setting the agenda in an increasingly important market, on the verge of disrupting an established market, or creating an entirely new market.” What did TR see in Crowdcast?
“TR saw how we were executing on bringing a very innovative enterprise software to market. It really is the third category – creating an entirely new market. Crowdcast is very different from traditional forecasting systems and directly addresses some major issues in the corporation (namely organizational biases reflected in forecasts of key metrics). It’s a disruptive technology, bringing transparency and change to our clients.”
Chief Scientist Leslie Fine has some meaty credentials in market and mechanism design, with a Ph.D. from Caltech and 8 years at HP Labs with BRAIN, an “information aggregation tool that harnesses all of the power and truth-telling properties of market mechanisms and implements it with all of the simplicity and robustness of a simple survey.” And your Board of Advisors includes academic heavies Andrew McAfee from MIT and Justin Wolfers from Penn. Can you share some of the behavioral, analytic, technical and business insights gained over the years from research that are now being leveraged in Crowdcast?
“Crowdcast's platform represents the team's more than 25 years of experience in mechanism design, corporate finance, business intelligence software, and collective intelligence implementation. We have incorporated a great number of insights, from those as straightforward as noting that 30% of our participation occurs after our weekly email goes out to those as complex as implementing a new and patented Bayesian market maker.
However, the biggest learning is that having first-class technology is only half the battle. Ensuring it is presented in a manner and time to get the best of the cohort is critical. Prediction markets and other theoretically efficient mechanisms have enormous potential, but are generally designed on the whiteboard and tested in the experimental lab. However, real business situations are full of behaviors, frictions, and dynamics not present in those environments.
Determining the right team to ask, the proper questions to ask of them, and how to reason about these answers in light of traditional data sources are key challenges. When Crowdcast integrates into the corporate BI systems, these challenges become trivial. The right questions are those currently being tracked, the cohort are those with the access, and the right presentation is as a double trend line alongside the traditional metric.”
Your business model revolves on “integrated SBI solutions – a SaaS software platform, consulting services, and on-going support – deliver timely visibility into the true state of your business.” Is this the crowd-in-cloud model? Can you give us an example of a successful soup-to-nuts customer? Is there an annuity revenue stream in your business model that'll support the growth investors demand? What about the OEM market where complementary products embed Crowdcast technology? Do you ever see Crowdcast licensing software to customers?
“The most common deployment is as SaaS, but we've also done on-premise licensing of the software. The pricing is based mainly on the number of seats. Our SaaS model has significant annuity revenue stream. We're talking to a PPM and a couple of BI vendors about OEMing our solution.
Hershey has been using our SaaS solution for over a year to forecast quarterly sales and other key metrics. The 'crowd' they're tapping is senior managers, director and above. In their most recent quarter, the 3 Crowdcast metrics forecast were all more accurate than the planned metrics.”
Where do you expect Crowdcast to be in 3 years? Is the company headed towards an IPO? Would I be better off asking this question in a Crowdcast market/game? :)
“The most likely outcome is an acquisition by an enterprise software company who wants to add a collaborative and engaging layer to their BI, forecasting or PPM platform. The idea of 'gamifying' enterprise software is really taking off, and Crowdcast is the only game-based platform that reliably delivers business metrics.”