Next Shift: From Big Data to Deep Data
It’s no secret that today’s consumer looks and acts differently than the consumer of fifteen years ago. While in the past, consumers may have been satisfied with standardized services and solutions that matched those of their neighbors, friends or acquaintances, people now expect products and services that best fit theirindividual needs and interests.
We have already seen this play out in multiple industries, perhaps most visibly in the telecom world. Thanks to smartphones and data networks, that industry has moved from a system of telephone poles and wires that delivered the same services to everyone, to one defined by personalized service and application delivery which allows consumers to use their phones in ways that best fit their lifestyles. Telecom companies actively leverage their consumers’ behavioral, transactional and demographic data to personalize their services and engage customers more deeply than they did just a few years ago. Today, these data-centric approaches to customer engagement are redefining other industries like energy, healthcare and retail which all rely on good data to stay relevant to their customers.
With the growing presence of big data, businesses are becoming more equipped to handle the tsunami of data about their customers. However, because of a lack of precise knowledge of the value embedded within this huge crush of data, many businesses have been stuck in the “data for data’s sake” trap capturing every piece of available information, a phenomenon I call the “data hoarding” mentality. With this approach, it’s easy to become overwhelmed by the amount of data available and get stuck focusing on the challenges to collect and store terabytes of information, rather than effectively leverage the information to deliver on the promise of personalized relationships and to add meaningful business value.
Here Comes Deep Data
As big data moves beyond hype to realized value, things are beginning to change. As we enter 2015, companies will move toward the "Deep Data" framework an approach based on the premise that a small number of information-rich data streams, leveraged properly, can yield more value than masses of captured data. By shifting to a deep (rather than big) data approach, businesses are able to better understand their customers and offer actionable, scalable and customized insights while crucially enhancing the value of the economic investment in data to their businesses.
It’s a concept we first saw in its infant stages during the 1980s, when the FICO score was introduced in the financial sector. The FICO score used a few rich sources of financial history (typically credit card activity history) to determine the credit-worthiness of a potential borrower. In the modern era, with plummeting data storage and computing costs, it remains a truism that fundamental value is generated by a structured approach to leveraging the right data for the right purpose using the right tools. This is the central thesis of the deep data framework.
In the energy sector, deep data-driven technology is helping utilities better engage consumers and benefit from trusted customer relationships in the face of new energy choices, like solar, microgrids and retail energy providers. As consumers become “prosumers” who create new energy options often at their own sites, utilities must find new, innovative ways to deliver value to customers who want the most cost-effective, energy efficient options. In response, utilities, like E.ON in the United Kingdom, are using software that takes a deep approach to data to unlock and parlay key energy insights to customers, helping them become smarter energy consumers. Similar examples abound in healthcare, retail and other industries.
Companies thus need to gear up on both the organizational and technical fronts to best leverage the huge amount of data available to them about their consumers. From the organizational perspective, companies must start to internalize and adopt a data-centric view at every level of the organization. This would mean, among other things, developing a vision and plan for leveraging the data, appointing a chief data officer, planning for data security and privacy, and democratizing access to data and analysis to everyone in the organization. This is not an easy task, but a necessary one.
On the technical front, a successful deep data company would meld and leverage three core elements: domain expertise (expert knowledge of the specifics of the business), data science (a set of specialized computational and mathematical skills that enable scaling of data-centric insight), and the right IT infrastructure (typically, cloud-based services like Amazon Web Services). Companies typically have in-house experts available, but do not always leverage their deep expertise fully by abstracting their specific insights to build models of consumer behavior. Getting these resources together with the technical folks is a key element.
The second resource, data scientists, is an emerging breed of specialists trained to leverage modern computing and mathematical knowledge in the context of specific business problems. A famous example of the outcome of data science is Amazon’s recommendations based on previous purchase history that is often uncannily accurate. Finally, of course, is the fantastic amount of cloud-based infrastructure that makes it possible to massively scale at low cost the insights originated from the combination of domain experts and data scientists.
Regardless of industry, deep data is poised to disrupt the way organizations engage customers. Those that embrace the deep data model will see success in sales and customer engagement, while those that shy away from analytics will fail to meet consumer demand ultimately failing to compete in the fast paced and fast changing world. As data-savvy organizations rise to the top, we can expect more options and opportunities to make well-informed spending decisions in 2015.