Managing the Data Explosion

As IT becomes ever more prevalent in nearly every aspect of our lives, the amount of data generated and stored continues to grow at an astounding rate. According to IBM, worldwide data volumes are currently doubling every two years. IDC estimates that 45GB of data currently exists for each person on the planet: that’s a mind-blowing 281 billion gigabytes in total. While a mere 5 percent of that data will end up on enterprise data servers, it is forecast to grow at a staggering 60 percent per year, resulting in 14 exabytes of corporate data by 2011.

A major trend over the last few years has seen many organizations implementing enterprise resource management and customer relationship management solutions.  This, in turn, has caused a dramatic increase in the amount of data we are storing about our customers, prospects, partners and suppliers.  

Companies are also investing in ever more sophisticated business intelligence and analytics.  In an increasingly competitive marketplace, the ability to base business decisions on solid, reliable and timely management information is becoming a key differentiator, but trend analysis can require very large amounts of historical data to be stored and managed. 

The trend toward company consolidation is not a new one, but the current economic situation has inevitably resulted in a significant increase in the number of mergers and acquisitions. This is creating a huge increase in data volumes, with associated data duplication and application retirement issues. Organizations are faced with not only managing all of their own data, both historic and current, but also this influx of additional data from other parties.  Imagine the data headache of combining all of the ERP, CRM, BI and analytic systems from different organizations into one manageable enterprise system.

Corporate compliance legislation has had a major effect on how we use, store and maintain our data.  The requirements placed on organizations by HIPPA, Sarbanes-Oxley, Basel II and others mean that many companies are having to keep hold of more data for longer periods. Just as importantly, that retained data rapidly transforms from a corporate asset to a liability once the legal minimum retention period has expired, making it vital that such data can be accurately identified and deleted. It is vital that organizations adhere to this legislation in order to avoid the cost of court appearances, heavy fines and the resultant damage to the brand.

New capabilities within the databases used to store corporate information are another major driver of data growth.  For example, DB2 now supports XML and LOBs (“large objects” such as audio, video, images, etc).  The ability to store this kind of data alongside more traditional structured information can be very useful but can also have a huge impact on the overall size of the database. Other technical trends that are contributing to database growth include storage of data in Unicode format (which can often expand overall database size by 10 to 50 percent depending on the data), and duplication of databases due to replication requirements and/or backup strategies.

Finally, there’s the perennial problem of removing old or obsolete data once it has reached the end of its useful life. Application data archiving is often considered as an optional extra, and even if it is included in the initial project plan it is often the first item to be postponed to a later release.

This unprecedented growth in data volumes is having a significant effect on many organizations. Perhaps the most obvious impact is on operational costs.  More staff time is required for routine maintenance and data-related exception handling such as out-of-space conditions and repartitioning.  As the database increases in size, so too does the central processing unit cost of running batch operations and routine housekeeping.  Ongoing running costs also increase due to the additional disk space required, and storage and processing capacity upgrades may be needed even though they often haven’t been budgeted for.  

Painful though they may be, increases in operational costs aren’t the end of the story. What price can you place on customer satisfaction? Performance for critical application processes can degrade as data volumes increase, resulting in missed service level objectives. Teams across the whole organization may be affected, with call center staff unable to access the information they need quickly enough to satisfy customer demand.  

Various coping strategies are available to address the issues associated with rapid data growth. Measures such as implementing database partitioning and data compression or purchasing extra CPU/direct access storage device can help.  However, these have their own costs there are many issues still remaining, including:

• Disaster recovery times,
• Legal risk of exceeding minimum data retention periods (data as a liability, not an asset),
• DBA effort to manage/tune workloads and databases and
• Cost of spending IT budget on maintaining current capacity, not innovating.

So, what are the alternatives? Implement a data archiving strategy!  

According to a recent Gartner report, database archiving significantly lowers storage costs for primary storage by moving older data to less costly storage. They go on to say that archiving reduces the size of primary storage, resulting in improved application performance and lower storage requirements for copies of the database for testing, backup and other purposes.

Also, you may think that archiving is only applicable to the largest of applications, but in the same report, Gartner states that performance and cost improvements can be sizeable, even with applications that have less than 200GB of data.