Many of us who have taken a psychology course or two may remember Abraham Maslow’s groundbreaking theory developed in the 1940s involving the psychology of human motivation and emotional development. Before Maslow expanded it in later decades, the original pyramid included five stages in what is now known as the “Hierarchy of Needs” pyramid (shown below).  Notably, Maslow focused on what human beings got right rather than taking the more historical path in psychology which played up the more dysfunctional, and, to be honest, fascinating aspects of human behavior.

Applying a Hierarchy of Needs Toward Reaching Big Data Maturity

Maslow’s proposition behind the “hierarchy of needs” model is that beyond the basic physiological needs of survival and security, most human beings are inherently motivated to grow and build on intellectual and emotional milestones toward achieving their full potential.  This model serves as a great example for enterprises that want to discover the full potential of their big data practices. The Big Data Maturity Model (shown below) takes this concept and applies a hierarchy of needs to global businesses’ aspirations to develop technology platforms as well as business processes. Regardless of where your business lands on the evolutionary scale of big data maturity, the key is to maximize potential at each stage and build on these tangible milestones to get to the next level on the big data maturity spectrum. The following stages offer companies a glimpse into where their business sits on the big data maturity scale, and offer insights to help these businesses graduate to the next level of big data maturity.

The First Stage - Discovery

This level is similar Maslow’s first stage of physiological development. Companies that reside in this evaluation phase are just beginning to research, review, and understand what big data is and its potential to positively impact their business. At the enterprise level, this business is beginning to realize it will need to accommodate growing big data storage requirements in order to keep pace with current technology. To graduate from this stage, it’s important to understand that POV use cases can be applied. Getting to the next level also requires seeing big data as more than just data storage or Hadoop.

Key traits for businesses in this phase: 

  • People – a few people may be trained or knowledgeable on big data or in relational databases. Data best practices not defined and not being created;
  • Process – no process at corporate level, and no routine best practices for data and databases;
  • Technology – relational databases, sub 1 TB level, data growth is not planned or handled with current assets and technology.  Applications will not handle higher level data growth beyond current data sizes.

The Second Stage – Startup

Businesses in this phase continue to learn and understand what big data entails.  Now, however, similar to Maslow’s second stage involving safety and security, they are taking steps to design, plan, and install an initial cluster (Hadoop, MPP database, for example), evaluating and selecting the correct hadoop distribution and toolsets that are appropriate. Experts review analytical use cases and obtain the appropriate data to support these use cases. At the enterprise level, the business takes steps to plan and architect its big data storage requirements. The business realizes use cases and analytics that can applied, and goes beyond seeing big data as data storage or Hadoop only.

Key Traits for businesses in this phase:

  • People – several staff who are trained or knowledgeable on big data and its initial concepts are researching and defining big data best practices.  Staff are also getting trained in Hadoop or  MPP databases and beginning to come up to speed on technology;
  • Process – the business defines a corporate level process for big data architecture and systems, training, and certification;
  • Technology – the business reviews the architecture, design, and installation of Hadoop and  MPP databases in the technology space, along with integrating EDW and other enterprise level data repositories. Obtaining storage that will scale beyond the 1 TB data storage space. Applications are being designed that can use the scale and performance of an MPP system.  

Stage 3 – Tactical Adoption

Description – This stage parallels Maslow’s third hierarchical stage involving love and acceptance. A business in this space understands the value proposition of big data, but is now learning what big data offers on a more tactical level. No strategic level planning or implementation has occurred at this stage. At the enterprise level, this business realizes its big data storage requirements are growing and is now looking at how to profit from that data as an asset. Milestone achievements mean selecting use cases to implement as a foundation for future improvements, and integrating to EDW and other data repositories. 

Key Traits for businesses in this phase:

  • People – additional people are being trained and learning big data technologies. Big data best practices are being defined but not yet implemented; 
  • Process – beginning to implement processes for big data implementation at corporate level and establishing best practices.  Maintenance for databases and data management;
  • Technology – Hadoop and MPP databases and their platforms are already installed with new implementations continuing at the tactical level.  Data sizes may be growing beyond the 1-10 TB level. Data is currently integrated from the EDW and being retrieved from databases into Hadoop.  Data growth is being planned with new storage assets and technology. Applications are being designed that can use the larger data amounts through parallelization (Hadoop and/or  MPP databases).

Stage 4 − Strategic Integration

Description – In this phase, which coincides with Maslow’s stage involving self-worth, a business understands and has implemented big data.  An enterprise at this juncture is realizing returns and positive ROI on the value proposition, with the majority of its use cases already developed and implemented.  At the enterprise level, this business fully realizes its big data storage requirements and has proactively planned for growth.  In addition to big data capabilities, (marketing, and competitive intelligence) analytics as well as predictive analytics have been developed and are being used for fully predictive capabilities in the identified use cases. Hadoop and/or MPP database clusters have been created, implemented and connected to data sources. Data from structured, semi-structured, or unstructured data sources integrated and transferred from EDWs, databases, log files, and machine data files have been used.

Key Traits for businesses in this phase:

  • People – many staff are trained, with hands-on experience working with big data (Hadoop and or MPP databases).  Overall, staff are trained on and following the data best practices;
  • Process – processes and best practices are fully defined at the corporate level, and most are  routinely followed;
  • Technology – big data technologies such as Hadoop, MPP databases, and analytics, are fully implemented and running on several projects and use cases.  Additional use cases are being created and implemented.  Data storage technology has been installed and is currently being used.   Applications are using these created MPP capabilities to take advantage of the big data technology. 

Stage 5 − Visionary Optimization

Description – In keeping with Maslow’s self-actualization phase, achieving this top level means that an enterprise possesses a visionary and optimized understanding of big data, and is maximizing it to realize full benefits. Big data capabilities are based on fully predictive use cases. At the enterprise level, this business has planned for and fully realized its big data storage requirements, implementing hardware and software to handle large amounts of data (500 TB to >PB scale). This business has created, implemented use cases and is routinely maximizing (enterprise-wide) big data and predictive analytics capabilities. Businesses are now fully transformed into a “predictive enterprise” where results are generated that allowing the businesses to predict as opposed to react to customer events.   

Key Traits for businesses in this phase:

  • People – many people are trained or knowledgeable on big data or in relational databases. Data best practices are fully defined and adhered to. Business is proactive, in terms of people resources being trained, and visionary in looking at technology training needs;
  • Process – best practices and big data processes are fully defined at the corporate level, and are routine for master data management purposes;
  • Technology – at enterprise level, Hadoop, MPP databases, EDW, and other technologies are fully connected and integrated into and fully support the business use cases for full ROI realization. Relational databases are replaced with Hadoop, MPP databases, “data marts” and “data lakes” and are now are at 100s TB to petabytes levels.  Data growth is fully planned and realized with current assets and technology.  Applications can run against higher level data amounts and are parallelized for growth.

It’s been said that Maslow believed only one in a hundred people ever actually achieving the fifth level of self-actualization. Conceptually, at least, the Big Data Maturity model is quite similar to Maslow’s motivational model, comprising a five-stage, evolutionary scale. Do the similarities between the two models correlate to fewer businesses achieving full integration in the big data sphere? There isn’t enough empirical data to support either statement, yet.  However, the benefits to achieving self-actualization, both personally and in business, so to speak, exist. 
The Big Data Maturity model helps your organization determine 1) where it currently lands on the Big Data Maturity spectrum, and 2) take steps to get to the next level. From initial discovery and start-up to more tactical adoption, strategic integration, and finally, visionary optimization, on what level does your organization view itself? What’s clear is that your business has the power to grow and build on its big data initiatives toward a much more effective big data approach, if it has the will. 

Steve Thompson is vice president of big data solutions and practice lead, CBIG Consulting.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access