Big Data: Too Much of a Good Thing?

Lauded as the next powerful driver of innovation, competitiveness and operational productivity, big data is increasingly being leveraged by forward-looking organizations of all sizes to gain a competitive advantage. Big data contains a virtual treasure trove of information about customers, trends and countless other valuable insights which may have the potential to transform businesses. However, in the race to unlock this promise, many enterprises now find they have more data than they are capable of handling. Here’s how CIOs, data warehouse managers and others can build the business case for data volume management to help them more effectively manage this data deluge.

Big Data’s Big Problem

An often-lost fact among the big data buzz is that a very specific strategy is needed to organize, manage, process and store the dramatic amounts of information being created on a daily – if not minute-by-minute – basis. Instead, priorities often skip directly to applying analytics. According to a recent Gartner survey cited in the Wall Street Journal , fewer than half of business leaders polled stated their big data strategy effectively addressed process efficiency, while just 37 percent cited benefits of cost reduction as a pillar of their plan.

As databases grow exponentially faster than at any other point in history, performance bottlenecks, decreased processing speeds and rising total cost of ownership are now mission- critical problems which many organizations must overcome. The good news is that proven best practices in data lifecycle management will provide a strategy to both tame the growth of data and harness this data to realize its true promise: real-time business intelligence. These steps include focusing on:

  • Archive strategy
  • Storage architecture
  • Retention management
  • Legacy decommissioning

Archiving Strategy

The first step toward sustainable data management is to develop an archiving strategy. For many, the idea of archiving conjures images of data locked away in a far-off vault, totally inaccessible for routine business needs such as customer service, customer acquisition, accounts payable/receivable or audit responsiveness. With a proper archiving strategy that includes transparent access to and retrieval of data, the business gains a key benefit: visibility.

To address archiving, begin with a business assessment that determines access needs and retention requirements for all data. Mapping data and filtering its importance when compared to the short-term and long-term business goals and legal requirements will paint a clear picture of which data sets carry high value versus low value, as well as its value at any given point during its lifecycle.

Such an assessment will illuminate where data originates, how it is utilized for business objectives, growth patterns and interrelationships with other types of information. This allows the business to align information value with an appropriate IT infrastructure effective in both the present and future. In essence, a natural blueprint for archiving will begin to emerge.

Storage Architecture

For most types of data, value typically diminishes over time and does not justify the investment in space on an active production database. By moving lower-value information to a lower-cost storage tier built with either disk-based storage, nearline or cloud-based storage-as-a-service, processing speeds on the database will improve and TCO will be reduced. The key step, however, is using an archiving architecture that ensures transparent access to and retrieval of the data.

Yet, cost and performance are not the only benefits of a sustainable data management strategy. Retention management and legacy system decommissioning polices are equally critical.

Retention Management

Businesses in all industries face increasing pressure from both internal and external compliance requirements. Imagine responding to an audit request with a short timeline. For many, this requires finding the proverbial “needle in an ever-expanding haystack,” with very little time to search. The same is true when requested data is part of a pending legal matter.

Regulations related to personal identifying information also remain in an ever-changing state of flux, as regulatory entities at the state, federal and even international level continue to modify and strengthen rules for how this data is stored. Failing to properly encrypt, mask or otherwise hide payment card information or the mountains of other personal data being saved poses a serious security liability which can be subject to large fines. A complete understanding of when data has reached the end of its life, and the policies needed to properly destroy it, are equally as important.

A comprehensive retention management solution both combines business-critical process data and documents into a controlled framework and implements standard lifecycle management policies so organizations can more quickly respond to audits and effectively manage the retention and end-of-life disposition of documents and data.

Legacy System Decommissioning

Legacy systems can be a bit trickier, and no two situations are exactly alike. Many companies keep multiple systems running just in case they need access to the information in the future, or more basically, to comply with corporate, legal or mandated retention policies.

A Business Case for Archiving

Data is a major catalyst that drives 21st century business decisions. But, too much of a good thing – including data – can result in system overload. While many executives harbor grand visions of predictive analysis fueled by big data that will cause an exponential spike in sales, there remain very practical problems that must first be addressed.

Processing performance, infrastructure costs and an inability to access the right information at the right time can dramatically, and negatively, affect corporate operations. With pressure to operate at the “speed of now,” these unnecessary detractors can be overcome with a well-planned data volume management and archiving strategy. Not only will improved efficiencies be created, but overcoming these obstacles will help forward-thinking organizations lay the groundwork to pursue future big data objectives.

For reprint and licensing requests for this article, click here.