Under pressure to meet project deadlines and gain continued funding, some data warehouse projects plan for completion rather than success. Business sponsors make their future support decisions soon after trying to use the data warehouse to find answers to real questions, and many practitioners are familiar with the pain associated with a warehouse that fails to meet expectations. Now, however, we are learning about the pain associated with overwhelming success more users, demands for new data feeds, slower performance, increased complexity of queries and the stress (both technical and personal) that occurs during acquisitions and mergers. This article reviews an architectural approach to a server- and network-neutral infrastructure that can support both rapid data warehouse growth and continued change in external systems. The phone rings and, in less than a minute, all the plans for the future of your data warehouse are a distant memory. You have been acquired or worse, you have acquired a company even larger than your own. Not only will the query rates more than double, but so will the number of feeder systems at least for the next 18 months.
What looked like a slam dunk data warehouse success two weeks ago now looks like a disaster waiting to happen.
With the acquisition, you expect early retirement programs and layoffs. Your most experienced DBAs and operations staff, the ones you will depend upon during the transition, are those most likely to leave. You now own "new" technologies servers, networks, databases that you had spent the last four years weeding out of your portfolio. Your disaster recovery plan, or your plan for a disaster recovery plan, is just that a disaster.
And if that isn't enough, the data warehouse must be available eight extra hours each day to support the managers, analysts and consultants planning the business combination. Once the acquisition is complete, complex new data feeds will have to be incorporated in record time; and you know these new users will pose more complex queries than your current clients.
This isn't a fantasy or someone else's nightmare. It could be yours. Today's business headlines are full of stories of mergers and acquisitions Citibank and Travelers, Chrysler and Mercedes, GTE and Bell South. In fact, the last 12 months have seen a record-breaking number of corporate marriages.
Mergers and acquisitions represent extreme change, but the need to move quickly into new markets to meet or anticipate competitive pressures is a daily reality. What can you do, today, to establish a server-, network- and operating system-neutral infrastructure that can survive dramatic change and growth? How can you put in place a technical architecture that enables you to make a radical shift in priorities and data warehouse design in order to support a new business mission?
One answer is to take a fresh look at enterprise storage as a core part of your infrastructure design. Enterprise storage isn't a box. It is an architectural concept that is particularly appropriate for data warehouses, data marts, operational data stores and the enterprise applications that feed them.
The following characteristics of enterprise storage should be considered when constructing the underlying architecture:
- Non-disruptive data backup/restore/ extract/load for ERP and data warehouse systems;
- Management of production, ERP and data warehousing storage resources from a central point of control;
- Data movement between ERP and data warehouse systems that does not impact servers or networks;
- Common disaster recovery for ERP and data warehousing systems;
- No constraints to growth and change; and
- Re-deployable storage assets among mainframe, AS400, UNIX and NT platforms.
Assume your architecture includes an enterprise data warehouse, an operational data store, a number of data marts and a variety of legacy and enterprise applications such as SAP, PeopleSoft, Oracle Applications or Baan. What is different about an infrastructure based on enterprise storage? In a nutshell, it is robust, flexible and reliable.
Support for Growth and Change
One exercise used to test the infrastructure design for robustness and flexibility is to walk through an imaginary merger with a company equal in size to your own. In data warehousing, data and query growth rates of 30-50 percent are typical. In an acquisition or merger, however, standard planning assumptions are invalid. By walking through an imaginary period of sudden rather than gradual growth, potential flaws in scalability, disaster recovery and operational procedures become apparent.
During a merger, growth plans will be tested to their limits. Backup and batch windows are suddenly too small, network bandwidth is overtaxed and processor cycles are at a premium. How can you get more capacity quickly and absorb new technology?
More Information to More People More Often
Many of your new users may be in remote time zones, causing an overnight change from a 5 x 18 operation to 7 x 24. Where does the extra processing time come from as batch windows close?
Designs using enterprise storage allow multiple mirrored copies of the data to be "spun-off" quickly so backup can be done from a copy of the data without impacting access to the warehouse or to production systems. Additional processors (from the same or different vendors) can be attached to the enterprise storage system. Additional batch reporting streams and query processes can be run against copies of the data.
Using enterprise storage, data can be moved between legacy or ERP systems and the data warehouse, and between the data warehouse and data marts within the storage subsystem, not over the network. Transfer speeds are substantially faster, the network is not taxed and no processor cycles are used for the copy.
Data can be extracted from production systems and loaded into the data warehouse using mirrored copies as well. QA testing, software upgrades and other operational activities now occur on the data copies while the live data remains in production. Some companies have recovered up to eight additional production hours each day using these techniques.
Manage with Limited Staff
When the combined systems are examined during a merger, the staff dedicated to managing storage on each unique vendor platform will be found to be substantial.
Enterprise storage, however, continues to be managed through a central process using a familiar interface even though the processors attached to the storage may grow and change. The mechanism for establishing RAID, operational and backup policies is common to all data warehouse and transaction systems. Backup of all systems that share enterprise storage can be done as a single activity with multiple streams.
With IT staff at a premium, simplified operational management of storage can relieve a major pressure point, and allow IT staff to focus on business-critical applications.
Consolidate Data Centers
There likely will be an opportunity to combine data centers supporting data warehouses during a merger. The remote copy facility of enterprise storage automatically keeps additional copies of critical data accurate up to the second, even when the remote copy is miles away. This provides disaster recovery and has a side benefit that enables the easy migration of data from one data center to another with little interruption.
To move data centers, you can connect enterprise storage systems in each center via fiber or high-speed lines. The center to be moved (the source) treats the target or destination system as a disaster recovery site. The data is mirrored to the target site, and after a start-up period the mirrors become identical twins. The data warehouse applications then can be tested and cut-over to the target site when convenient.
Recovering from a Disaster
Data warehouses have become mission critical and, as such, have acquired zero tolerance for down time. While tape is a critical component for a complete data security plan, the process of recovering terabytes of data from tape alone is lengthy and prone to both physical and human error. Enterprise storage allows for the rapid recovery of a data warehouse including all the optimizations, indexes, shortcuts and temporary tables. The data warehouse and the transaction applications can be back on-line in less time than it takes to find and load a single tape.
Preparing for Change
Whichever server technologies you've chosen, in a merger/acquisition the choices will be challenged. New technologies will emerge. Equipment from different vendors will be available as obsolete applications disappear.
Enterprise storage enables you to preserve the investment in existing storage. Storage can be reallocated from decommissioned mainframe applications after Y2K to feed a growing data warehouse on UNIX. Storage from overloaded UNIX applications can move to NT to support expanding data marts.
A New Architectural Component
Enterprise storage is now more than a necessary IT component. It is a key architectural element for both data warehousing and transaction systems that allows rapid response to sudden growth and change. It creates an infrastructure that encourages the adoption of new technologies. It eases management headaches.
You may not expect to acquire or to be acquired, but why wait until it is too late; take the imaginary walk-through today and prepare your data warehouse for whatever changes the future may hold.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access