Big data and Hadoop (Yellow Elephants) are so synonymous that you can easily overlook the vast lanscape of architecture that goes into delivering on big data value. Data scientists (Pink Unicorns) are also raised to god status as the only real role that can harness the power of big data -- making insights obtainable from big data as far away as a manned journey to Mars.

However -- as I recently participated at the DGIQ conference in San Diego and colleauges and friends attended the Hadoop Summit in Belguim -- it is apparant that organizations are waking up to the fact that there is more to big data than a "cool" playground for the priviledged few.

The perspective that the insight supply chain is the driver and catalyst of actions from big data is starting to take hold. Capital One, for example, illustrated that if insights from analytics and data from Hadoop were going to influence operational decisions and actions, you need the same degree of governance as you established in traditional systems. A conversation with Amit Satoor of SAP Global Marketing talked about a performance apparel company linking big data to operational and transactional systems at the edge of customer engagement and that it had to be easy for application developers to implement.

Hadoop distribution, NoSQL, and analytic vendors need to step up the value proposition to be more than where the data sits and how sophisticated you can get with the analytics. In the end, if you can't govern quality, security, and privacy for the scale of edge end user and customer engagement scenarios, those efforts to migrate data to Hadoop and the investment in analytic tools cost more than dollars, it costs you your business.

Unfortunately, vendor tools in data management capabilities to make data meaningful and oriented for ease of analysis and use, trusted, and secure is still in its infancy for big data. Without rapid improvement, application developers will be stymied in their ability to create evolutionary, innovative, or disruptive capabilities that make their organizations competive.

It's time to demand more than rapid development of features to piecemeal gaps in our big data management and governance needs. Or be satisfied with early releases that take time and significant skill to deploy.  For big data to take hold, it needs to fit the objectives, skills, and agility requirements of more than the 1%. It is only then that we will have a true data (INFORMATION and INSIGHT) democracy.

Get acquainted with the patterns of data governance for big data.  Do Data Quality The Big Data Way

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access