Last week I wrote about the perils of “common sense” thinking for business skillfully articulated in sociologist Duncan Watts’ book “Everything is Obvious: *Once You Know the Answer – How Common Sense Fails Us.”
Watts’ thesis is that the common sense that generally guides us well for life’s simple, mundane tasks often fails miserably when decisions get more complicated. A host of individual biases and irrational heuristics, social emergence complications and historical blindness conspires to make us much less the cool scientific actors than we’d like to presume. Watts’ remedy? A healthy dose of evidence-based, “uncommon sense” that he outlines in part two of the book.
So, ever the scientist, Watts chooses analytics versus the experts as the uncommon sense solution for business, right? Not so fast. The author’s suspicious of the “high modernists” who 90 years ago posited “the business of running an entire economy is within the scope of ‘scientific’ planning.” Further, the fatal flaw of the idea that laws of nature, applied to appropriate data, could be used to predict the future – Laplace’s demon – is that what may work for predicting simple processes fails for the more complex. Alas, the effects of marketing campaigns or outcomes of strategic plans are indeed complicated.
Watts sees other problems with the dream of prediction, too. Humans have difficulty thinking in the probabilistic terms about the future implied by predictive analytics. And how does PM handle once-in-a-lifetime or even non-recurring events? What about “black swans” that reside in the extreme tails of probability distributions?
Hurricane Katrina wasn’t even the biggest storm of its summer. What made Katrina a black swan had more to do with the levee failures in New Orleans, the subsequent flooding, the displacement of the population and the ineffectiveness of the emergency response than the storm itself.
Finally, even when events are predictable, there’s still plenty of randomness to go around. Is legendary mutual fund manager Bill Miller’s fifteen year run of beating the S&P 500 index, followed by dismal performance in five of the next six years, evidence of luck or skill – or neither?
Though wary of the claim that predictive analytics is the solution to all the world’s ills, Watts is still a big proponent for events that “conform to a stable historical pattern.” As much as statistical algorithms, he touts the prediction prowess of electronic markets that consolidate the wisdom of crowds. For both PM and opinion inquiries, according to the author, simple models perform indistinguishably from complex ones, and ensembles that average many predictions – either individual or analytic – are preferred.
Like most of his business academic peers, Watts is no fan of traditional, top-down strategic planning. He cites economist William Easterly on the failures of heavily-planned foreign aid programs, proposing instead agile trial and error “searches” for success. “A Planner thinks he already knows the answer … A Searcher admits he doesn’t know the answers in advance … and hopes to find answers to individual problems by trial and error.”
Watts’ searching over planning mentality proposes quick-hit, measure-and-react approaches to strategizing that rely less on making error-prone predictions about big trends and more on reacting quickly to change. Internet fashion retailer ModCloth outlined such a strategy for both buying and managing inventory at the recent Strata conference in Santa Clara.
The searcher’s tools? Agility, field experiments, open innovation, crowdsourcing, data and analytics. The author lauds the science of business manifesto outlined by MIT professors Erik Brynjolfsson and Michael Schrage for its potential of bringing hypotheses, experiments and analytics to all facets of business.
Watts is also a big proponent of what Google economist Val Varian calls “predicting the present,” a data science methodology that forecasts near-term behavior of interest, such as unemployment or demand for a new camera, by correlating trends from related Google searches. Varian deftly demonstrated the potential of predicting the present in a keynote at Strata.
At the end of this very enjoyable read, Watts struggles to reconcile a vocation as a serious sociologist to his position as data scientist with Yahoo!. Give him credit for eating his own dog food: rejecting “physics envy,” big theory, common sense sociology for “middle-range,” practical inquiry enabled by the data deluge – and fueled by the uncommon sense of learn-as-you-go planning, field experimentation and analytics.