Top 10 Priorities for Big Data Management
Sure, big data can be overwhelming. To help simplify the conversation within your organization, here are 10 priorities for big data management – care of TDWI Research.
“The first step toward value is to manage big data. If you don’t capture, store, process, and deliver big data, you won’t have it to repurpose for valuable applications in customer intelligence, operational efficiency, business monitoring, and so forth,” said TDWI. “Eighty-nine percent of survey respondents say BDM is an opportunity—but only if you seize it. Know the common paths to business value and follow them. The primary path to business value from big data is discovery analytics. A second path joins new big data with older enterprise data to extend complete views of customers and other business entities. A third path taps streaming big data to enlighten and accelerate time-sensitive business processes.”
“Streaming big data can enable business monitoring applications, and big data can expand the data samples that data mining and statistical analysis applications depend on for accurate actuarial calculations and customer segments or profiles,” TDWI said.
“The focus should be on training and hiring data analysts, data scientists, and data architects who can develop the applications for data exploration, discovery analytics, and real-time monitoring that organizations need if they’re to get full value from big data,” said TDWI. “When in doubt, hire and train data specialists to manage big data, not application specialists. Most BI/DW professionals are already cross trained in many data disciplines; cross train them more.”
“Due to big data’s diversity, diverse technology teams will need to play coordinated roles,” said TDWI. “As a business asset, big data should be managed for broad access and leveraged by multiple business units and stakeholders. It takes a lot of collaboration to be sure everyone knows their role and has their needs met.”
“After all, the goal is to integrate big data into your well-integrated enterprise data and BI/DW environments, not proliferate 21st-century spreadmarts,” said TDWI. “Someone (probably not you) should decide whether big data platforms will be departmentally owned (as a lot of analytic applications are) or shared enterprise infrastructure supplied by central IT (similar to how IT provides SAN/NAS storage, servers, the network, etc.). Even if big data management begins in a silo (and you do have to start somewhere), make integration with other enterprise data management systems a second-phase priority. To make the integration happen, look for big data platforms (both open source and vendor built) that enable the integration points.”
“For example, an obvious place to start is to rethink the data staging area within your data warehouse,” said TDWI. “That’s where big data enters a data warehouse environment and where it is usually stored and processed. Consider moving your data staging area to a standalone big data management platform (on Hadoop, a columnar DBMS, or an appliance) outside the core data warehouse. Architecture can enable or inhibit critical next-generation BDM functions such as extreme scalability, complete views, unforeseen forms of analytics, big data as an enterprise asset, and real-time operation.”
“For one thing, big data management is, more and more, a multi-platform solution (as are most data warehouse architectures), so you should expect to further diversify your software portfolio accordingly to accommodate big data fully,” said TDWI. “For another thing, survey data shows that the software types poised for the most brisk new adoption in the next three years are Hadoop (including HDFS, MapReduce, and miscellaneous Hadoop tools) and complex event processing (for streaming real-time big data). After those come NoSQL DBMSs, private clouds, and data virtualization/ federation. If you’re like most organizations surveyed, all these have a potential use for your BDM solution, so you should educate yourself about them, then evaluate the ones that come closest to your BDM requirements.”
“There are many types to consider, including relational DBMSs, columnar DBMSs, data appliances, and other engineered systems, as well as Hadoop, NoSQL DBMSs, and the many other options,” said TDWI. “However, matching your BDM requirements to vendor products isn’t always about the entire platform; it sometimes comes down to specific functions such as in-memory processing, in-database analytics, complex event processing, MapReduce, and robust interfaces to other data platforms.
“Create a plan for your BDM maturation process,” said TDWI. “You have to start somewhere, so start with relational data, then move on to other structured data, such as log files that have a recurring record structure. Carefully select a beachhead for unstructured data, such as text analytics applied to call center text in support of sentiment analysis. Look for mission-critical data that’s semi-structured, as in the XML documents your procurement department is exchanging with partnering companies. Then continue down the line of big data types.”
“The strategy needs to spell out a wide range of road maps, standards, preferences, and stewardship guidelines, depending on what your organization needs and has a culture for,” said TDWI. “For example, you could lay out a road map for maturing from structured to semi-structured to unstructured data, as noted above. Since big data comes in many forms, you need to determine the preferred platforms and interfaces for capturing, storing, and processing each form. You should design a workflow for managing big data forms in their original state, plus processing that into other states for customer intelligence, data warehousing, discovery analytics, and so on. Big data isn’t the storage problem it used to be, but you still have to plan capacity carefully, as well as related issues such as the acquisition and upgrade of data management platforms. Assuming you have an enterprise-scope data strategy and data architecture, you need to determine the many places diverse forms of big data should take in those. Finally, all the above must be supported by influential business sponsors (through stewardship and governance) so that big data management aligns with business goals for maximum business value.”
Simplify the big data conversation within your organization. Here are 10 priorities to get you started – care of TDWI Research.
The Top Companies for Integrated Systems Visionaries & Niche Players
The Top Companies for Integrated Systems Leaders and Challengers
Gartner's Steps for Creating the New Digital Ecosystem
Gartner Identifies Five Domains for the New Digital Platform
Gartner’s Top 10 Strategic Technology Trends for 2017
Gartner’s Top 10 Predictions for IT in 2017 and Beyond
10 Things Your Next Firewall Must Do
5 Things to Do When Your Organization Gets Hacked