Continue in 2 seconds

Four Golden Rules of Successful Data Migration

  • John Morris, Tony Sceales
  • July 23 2008, 12:55pm EDT

When implementing a new set of applications that better meet your changing business requirements or removing duplication and inefficiency in your IT infrastructure, you cannot avoid migrating your business-critical data. You know that while failure is not an option, industry failure rates for complex migrations are worryingly high. Perhaps it might help to know that successful data migrations share a number of common characteristics. This article outlines the four golden rules of successful migration.

Not all migrations are equal. Many migrations are fairly straightforward and motivated purely by technical or IT drivers. For example, at the storage level companies might need to migrate their data to cheaper forms of storage, and at the database level they might need to mirror data to another disk. At the application level, however, data migration can be very complex in large enterprises, because it needs to span both technical and business issues. According to Bloor’s Phil Howard, Forbes 2,000 companies already spend at least $5 billion per year on migrations, yet 80 percent of them still go over time or over budget.1

To understand why this is the case, it is important to recognize how complex IT infrastructures have arisen. Often, they mirror an enterprise’s history and have come about as the result of mergers and aquisitions (M&As), organic growth, product launches and new initiatives over many years. While individual parts of the infrastructure might be well architected, robust and functional, the overall structure might be poorly understood and far from optimal. However, while IT optimization might be desirable, unpicking this complexity is far from easy and risks catastrophic failure, such as lack of availability of business-critical data or failure of service or product delivery systems. However, analyzing successful IT migration projects reveals common success factors, as shown in Figure 1.

Rule one: Data migration is a business, not a technical, issue. Historically, migration has been viewed as a purely technical issue, and the most pressing concern has been how to migrate data. But because IT doesn’t necessarily own either the source/target applications or the data that the business uses to function, it doesn’t have the power or the necessary knowledge to deliver what is required by the business. Further, an increasing feature of the modern IT environment is that IT has come out of the data center and is no longer controlled by the IT department in the way that it might have been previously. Involving business managers in data migration projects is therefore essential. In fact, one of the most common characteristics of successful migrations is they are business led rather than technology led. Putting the business in the driver’s seat means that before we ask, “How do we migrate data?” we first answer a series of important related questions that help us frame and scope the project:

  • Why are we migrating data?
  • What data should be migrated?
  • When should it be migrated?

These questions cannot be answered by technicians. They can only be answered by business managers, which is one reason why it is essential for the business to drive the migration. Ensuring that the business makes the decisions and drives the project also frees IT to do what it is best at, which is handling the technical aspects of moving the data.
This suggests any technical solution to the migration must provide the business with clear visibility and control over the way the program is progressing.

Rule two: The business knows best. The second rule of successful data migration is closely linked to the first. The business drivers, not technical ones, should take precedence. It is critically important that business goals define the solution and approach selected, and not the other way around. The best technical solution is not always the best business solution. Therefore, to be successful, the chief business stakeholders must define their requirements and take responsibility for driving the project.

The business is also better placed to prioritize the migration, deciding which data to migrate first. For example, it might decide that migration be triggered by a customer selecting a new service that can only be supported on the new application stack. Or it might decide that the most important customers be migrated first, or second, or last according to business need. By allowing the business to drive the migration, an enterprise has a better chance of delivering the right solution for its needs and maximizing ROI.

However, IT must be aware that the business will not be able to predict at the outset what issues will surface during the migration. Moreover, in a long-running program they will absolutely have to handle changes in priority and direction. The chosen migration technology must be able to support such changes without restarting every time.

Rule three: No one needs, wants or will pay for perfect data. Applications are only as good as the data they have available to them. We also know that many data migrations have been scuppered by overestimating or not understanding the quality of the source data. Oh, the joy of legacy data with its gaps, inconsistencies and redundancies! While enhancing data quality is a worthy goal, it is really important not to go off on a tangent midproject in the quest for perfect data quality. Like overspecification of an application, the quest for data perfection can result in negative consequences for the project. It is where many, many projects run aground - inflating both the cost and the time to deliver the project. To avoid this trap, data owners and users need to determine the level of quality they require at the start of the project, in order that the technologists have an appropriate goal. Project managers must also be aware of the quality of their legacy data and allow adequate time and budget to achieve the requisite data quality.

A successful migration strategy will also need to incorporate a range of cleansing strategies at different points in the life of the program - sometimes premigration, sometimes in flight and sometimes postmigration - but always with a conscious decision from a business manager. Many historical migration approaches require bringing the whole process to a halt while a data quality issue is explored and resolved. A modern platform will provide the flexibility to continue migrating while this is done.

Rule four: If you can’t count it, it doesn’t count. Another challenge is how to measure data quality in order to assess the state of your legacy data and determine the level of quality your business users require. To make matters worse, data quality is not static; it erodes and improves over time. It is important that the measures you use make sense to business users and not just to technologists. This enables you to measure deliverables, perform gap analyses and monitor and improve ongoing data quality. It also ensures that you are concentrating your efforts on where business users see value and can quantify the benefits.

Reconciliation of the data migrated from source(s) to target(s) is always a critical activity - how do you know when you’re done? When dealing with dynamic environments where you can’t freeze the data, this becomes even more challenging; you need to be able to handle a shifting scope. A flexible data model and a closely coupled reporting capability are key to understanding and driving this process.

Achieving a Business-Driven Migration

Having put a business-driven migration project in place, the trick is to select methodologies and technologies that can deliver against these requirements. A business-driven migration involves decoupling the technical problem of moving the data from the business processes that use it. This requires a migration solution that enables you to easily encapsulate the business problems you face while being flexible enough to cope when those requirements change. This ensures that ROI from new applications is maximized and operations are enhanced rather than adversely affected.

The critical importance of getting the migration right from a business perspective was articulated at a recent British Computer Society meeting by BT’s Phil Dance: “Increasingly, our business case is going to depend on how good we are at getting our data across. A bad data migration ultimately means a bad customer migration, and in a competitive market, that’s very bad news.”2


  1. Phil Howard and Carl Potter. “Data Migration in the Global 2000 - Research, Forcasts and Survey Results.” Bloor Research, September 2007.
  2. Phil Dance. “BT PIC’s CIO for Technology.” Data Migration Matters, November 2007.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access