Solving the Big Data “Abandonment” Problem

Register now

The explosion of investment in big data continues, with global spend expected to reach $114 billion by 2018 according to ABI Research. This forecast includes the money spent on internal salaries, professional services, technology services, internal hardware, and internal software. This investment supports an overwhelming mandate from executives for their organizations to become more data-driven, not only to make better strategic decisions but to gain operational efficiencies that result in higher margins and ultimately benefit shareholders.

Given the strategic imperative from the c-suite to harness the power of big data, why do the majority of these projects fail? Gartner estimates the failure rate to be nearly 60 percent. Similarly, Capgemini finds that only 27 percent of executives believe big data projects succeed and of those only 8 percent are “very” successful.

If adoption of big data is not the problem, what is? It seems that turning adoption into value for an organization remains elusive. Often, organizations fail to see justifiable ROI from their big data investments because no clear blueprint exists for how to take a project from inception to completion with delivering value in mind. The two biggest issues are:

1. Lack of Alignment Between IT and Business Stakeholders

Summarizing the findings of a recent whitepaper, “…many organizations put the technology before the business problem they are actually trying to solve.” That almost guarantees a Big Data project won’t deliver all the value it could. By contrast, a strong partnership between IT and business owners goes a long way towards ensuring success. Business stakeholders own the strategy, understand the right questions to ask of the data, and know what decisions advanced analytics can support and the problems it can solve. IT delivers the infrastructure and tooling to enable business innovation. Or as Delaware North CMO Todd Merry puts it, “Marketing is the driver of the big data car, but it doesn't go anywhere without IT.”

Identifying the Right Data Assets

Big Data has become synonymous with Hadoop. Hadoop solves the Big Data storage problem, but creates another equally vexing one. As organizations “dump” more and more data into Hadoop file systems, which lack traditional information architecture, they frequently lose track of what those systems contain. So they can’t find the right data to analyze.

Plus, Big Data projects should include other enterprise systems such as EDW, CRM, CMS, and local file shares, as well as external data sources. Ignore these sources and a Big Data initiative risks missing its target—predictive analytics based on all the information relevant to the task at hand.

Marilyn Matz, CEO of Paradigm4, explains it this way, “The increasing variety of data sources is forcing data scientists into shortcuts that leave data and money on the table. The focus on the volume of data hides the real challenge of analytics today.”

Operationalizing Big Data Projects

There are four key pillars to ensuring a Big Data project succeeds, whether an organization is starting out or trying to salvage a Big Data investment gone wrong.

1. Staffing the Project: Remember Big Data is not an IT project; it’s a business project. You need to build a cross-functional group made up of IT data experts as well as those with business domain expertise—marketing, sales, finance etc. Also, consider the various roles that should be involved from data mangers and data scientists to BI analysts and business owners.

2. Identify the Right Use Case – Don’t boil the ocean. Start small and show success. Data and information silos across the enterprise will not disappear, so gain a quick win with an individual business unit such as marketing. Identify a single use case that demonstrates business value, for example driving higher upsell and cross sell potential. Agree on the success criteria of a POC, collaboratively select the right data types and sources for the project, know the questions you want to answer, and determine the end user experience needs. That way you can navigate the roadblocks that will inevitably arise as you go through the POC process. You’ll also learn at what scale you can leverage Big Data in your current environment before you need to consider infrastructure changes.

3.Establish Executive Governance and Process – Executive governance can be the difference between success and failure for any Big Data initiative. Executive governance defines the desired end state, reaches out to key stakeholders, and establishes procedures that should be followed on a recurring basis—weekly, biweekly, monthly—to ensure everything remains on track. Lock in key checkpoints and stick to a project plan to avoid scope creep. And stay ahead of any data privacy and data security issues by addressing red flags up front. Governance is especially important when using outside technical resources. It provides a forum where any constituency can “raise its hand” when issues arise.

4. Identify Critical Technology and Tooling – By now it should be clear that Hadoop alone cannot make a Big Data project successful. Think through all the technology and tooling that every step of the project requires. Forrester Research recommends integrating business intelligence and Big Data in a flexible hub-and-spoke architecturewith components such as:


  • Data Storage – Gather enterprise data and information into a flexible Hadoop or a Hadoop-like technology – the data hub of the architecture
  • Data Preparation – Cleanse and transform the data into the right structure for downstream use
  • Data Discovery Acceleration – Profile, identify, and unify all of the enterprise information with a self-service data source discovery system
  • Predictive Analytics – Feed the data into an analytics engine and build predictive models
  • Data Visualization – Create dashboards and reports with business intelligence and data visualization tools

Successful Big Data Strategies = Big Gains

Organizations that successfully compete on Big Data analytics create an agile ecosystem for business stakeholders that enables them to quickly access all the right data assets in context. An agile analytics ecosystem fosters innovation, productivity, and improved executive decision-making capabilities.

For example, in the oil and gas industry, analytics can reduce health, safety, and environmental (HS&E) risk. They can provide the business case for higher safety standards and pinpoint the reason for HS&E incidents. In healthcare, analytics can help clinicians make better decisions at the point of care while reducing the costs for unnecessary testing and ineffective treatments.

Whether your goal is more strategic resource allocation, increased market share, greater revenue, better operating efficiencies, or reduced costs, don’t abandon your Big Data aspirations.

Stephen Baker is CEO of Attivio.

For reprint and licensing requests for this article, click here.