Dark Data: B2B’s Big Data Challenge

Register now

It’s estimated that in 2013, the world will produce an astounding 4 zetabytes (or 4 million petabytes) of new data. But unlike B2C companies, which are  pressed to make use of massive quantities of unstructured consumer data, B2B companies face a big data challenge that goes well beyond volume.

The issue is this: Efforts to scrape and utilize data from external sources like the Web and social networks hold little value if the data can’t be matched with a company’s own internal data – much of which is “dark.”

Dark data simply refers to all the corporate data that is collected from a variety of sources, but that isn’t routinely used day-to-day operations. Essentially, the data sits  on a shelf unused and collecting dust.  

Gartner analyst Svetlana Sicular puts it this way: “Similar to dark matter in physics, dark data cannot be seen directly, yet it is the bulk of the organizational universe.” And this vast array of dark data is Big Data’s biggest hurdle.

Breaking Out of The Data Silos

At many enterprises, this includes data locked into various silos – such as CRM, ERP, marketing automation and other operational databases – creating separate islands of information. In order to get value from external data sources such as news,  government contracts,  patent activity and social media, the external data has to be matched against the company’s internal data set in order to identify meaningful trends and patterns.

However, when much of a business’ data is dark, this poses a special challenge. How do you match it with other types of data, or set goals and objectives for what the business should try and get out of the data, when you don’t even know what the data is? And further,  how do you justify the costs associated with analyzing the data or determine which data is important to analyze and which is not?
To help assess whether or not a company’s dark data has significant business potential, consider this set of questions:

  1. Does the data in question exist within the organization or is it external?
  2. How easy will it be to access the specific data source?
  3. How will this data help solve a business problem?
  4. How clean or reliable is the data?
  5. What special skills or technologies does analyzing this data require?
  6. How can this data be used going forward?

To begin to pierce the mystery surrounding a business’ dark data, integrating the company’s various data silos is a good first step.  This makes it much easier for data “consumers” to analyze, research and discover new patterns that can facilitate decision-making. Enabling the data residing in various corporate silos to be shared is a great example of how the power of big data can be leveraged without introducing any new data at all.
The Barriers to Data Integration

One obstacle that needs to be reconciled is that the differing formats in which the data from different sources is stored doesn't allow it to be easily shared. For example, a company name is a common way to match information from different sources. But that becomes difficult when the company is referred to by more than one name (a including, Inc., incorporated, corp., corporation, abbreviations, etc.). 

Equally important is transforming the data from different sources into a single, usable format. Think of it as “turning the lights on” for the data. This is a multistep process:  First, the data has to be cleaned, matched and merged. Second, the data may have to be reorganized or “pivoted” to better reveal useful trends and patterns. Third, the target system has to be populated with the transformed data. All these steps require discovery, design and review to ensure the maximum insight and value can be extracted from the data.

A third consideration has to deal with managing the changes that occur to the data over time. These incremental changes need to be tracked, and each data source may have a slightly different way of identifying those changes.

All together, the value of the data from the wide spectrum of sources is much greater than the sum of the parts. Once integrated into a workable whole, it can be used to provide new insights and to drive decisions, leading to new efficiencies and innovations. By bringing its data into the light of day, a business can illuminate new routes to market.

For reprint and licensing requests for this article, click here.