Big Data from McKinsey

Years ago, I didn't think business strategy and BI were on the same page. Strategists were MBAs from top 10 B-Schools, while BI analysts were technologists/statisticians from state universities.

Companies hired strategists to be the business experts, while they hired BI types to help them assemble information to support performance management. I often felt strategists were reluctant subscribers to analytics. Strategy experts had their place divining business direction; analytics had theirs assembling rear view mirror intelligence.

But that's changed now – for the better, I believe. Over the past five years, important business thinking from the likes of Ian Ayres (“Super Crunchers”), Jeff Pfeffer and Bob Sutton (“Hard Facts, Dangerous Half-Truths & Total Nonsense”) and Tom Davenport (“Competing on Analytics”) has concluded decisively that, in contests of experts vs analytics, analytics wins, hands down. If you can't beat them, join them, so I guess I'm not surprised that McKinsey Global Institute, the research arm of blue-blood business consultancy McKinsey & Company, has authored a comprehensive, 156 page report entitled: “Big data: the next frontier for innovation, competition and productivity.”

I was a bit skeptical as I started the read, but the preface convinced me that the research was analytically substantive and not just strategy-speak: data science champions Hal Varian of Berkeley and Erik Brynjojfsson and Andrew McAfee from the MIT Center for Digital Business were acknowledged for significant contributions.

The point of departure for the report is a definition of “big data” as “datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.” The driving thesis is that “data can create significant value for the world economy, enhancing the productivity and competitiveness of companies.” The time is now for productivity advances because “the scale and scope of changes that big data are bringing about are at an inflection point, set to expand greatly, as a series of technology trends accelerate and converge.”

How can big data create value? The report notes several ways:

1. Creating transparency – enabling, for example, the manufacturing sector to integrate “data from R&D, engineering, and manufacturing units to enable concurrent engineering ... (to) significantly cut time to market and improve quality.” This seems much like traditional data warehousing.
2. Enabling experimentation – “organizations can collect more accurate and detailed performance data ... to instrument processes and then set up controlled experiments … (which) can enable leaders to manage performance at higher levels.” Super-crunching equals analytics + experiments.
3. Supporting human decision making with automated algorithms – “decision making may never be the same; some organizations are already making better decisions by analyzing entire datasets from customers, employees, or even sensors embedded in products.” The statistical learning world continues to progress.
4. Innovating new business models – “The emergence of real-time location data has created an entirely new set of location-based services from navigation to pricing property and casualty insurance based on where, and how, people drive their cars.” This affirms Mike Loukides' assertion “that data science enables the creation of data products.”  

The ascent of big data will create new flavors of companies that aggregate and analyze industry information. Some of these firms will “sit in the middle of large information flows where data about products services … can be captured and analyzed.”  An example of this is clinical information providers in the health care industry that market performance benchmarking data to hospitals and clinics.

Big data levers will also create opportunities for organizations to improve efficiency and effectiveness. To capture that potential, though, organizations must adapt their processes to an evidence-based foundation. “… the use of big data can enable improved health outcomes, higher-quality civic engagement with government, lower prices due to price transparency, and a better match between products and consumer needs.”

MGI notes disparities in the different sectors of the U.S. economy in the use of big data. The computer/electronics and information sectors, not surprisingly, are leaders in productivity with big data. And finance/insurance are on the cusp of major benefits – provided they can overcome technical barriers. On the other hand, the public sector and education are challenged by the “lack of a data-driven mind-set”, while health care “faces challenges given the relatively low IT investment.”

Good news for both data scientists and evidence-based managers: there's a paucity of workers needed to take advantage of big data. Big data … estimates a shortage of deep analytical talent of between 140,000 and 190,000 positions by 2018. More, the report “projects a need 1.5 million additional managers … who can ask the right questions and consume the results of analysis of big data effectively” in the same time frame. I've already sent a copy of the report to my college-age kids.

At the same time, there are knotty issues to be addressed before the full potential of big data can be realized. Data policies that consider privacy, security and the legal implications of intellectual property must be addressed. Technologies that support the management, integration, analysis, visualization and consumption of big data must continue to emerge and progress. And organizations themselves – from senior executives to big data practitioners, through business processes and incentives – must accommodate to the new opportunities.

Successful big data organizations will be those that recognize and act on the opportunities – and threats – presented. They will be creative in determining which pools of data can be combined to divine optimal data products. They will invest in IT capabilities as “necessary to capture big data opportunities relevant to their enterprise.” And they will attract employees who have the requisite skills for a big data world.

I enthusiastically recommend “Big data …”  to those of business, technology and analytics persuasions. While the report is a substantial read, I believe the time investment will more than pay for itself. Those, like me, unable to work through the entire manuscript in one sitting shouldn't be deterred. The 13-page Executive Summary provides plenty an excellent summary of the findings and recommendations.