Data migration involves converting data from one format and/or system to another, newer format and/or system. Migration projects do not occur frequently, so IT staff are often unprepared for the task, which is much more complicated than they first think. Adding to the challenge is that the focus is typically on the system or software migration (e.g., migrating to an Exchange Server) with not enough attention paid to the data migration (e.g., converting e-mail data where necessary for the new platform).
Prevent project delays and data loss by developing and following a thorough data migration plan that includes appropriate resources. Ensure critical data has the focus it deserves.
Migration Programs and Services
Commercial software for data conversion is available. However, if the data or system is highly customized, commercial programs are likely not going to work and migration programs will need to be developed in-house.
Identifying Data Interdependencies
The migration programs and testing must account for all applications and processes that interface with the data to ensure they are not affected by the migration. To identify these interdependencies:
- Assign business managers and end users to create a flowchart of the overall data workflow and identify all applications and processes that touch the data. Their familiarity with how the data is used will enable them to provide good detail.
- Use IT staff to identify back-end or technical interdependencies not covered by the business managers and end users.
The above process is similar to developing a business process model, which is essentially a flowchart or outline of the processes and workflows in the organization.
- Review the dos and don’ts before beginning. Refer to the Mclean Report research brief, “Dodge These Five Data Migration Disasters,” which outlines potential pitfalls as well as key steps to include in the migration plan. Also, refer to the Mclean Report research brief E-Mail Migration Headaches: Watch and Dodge.
- Prepare for the data migration project:
- Assign responsibilities for the project. Include a project manager, programmers to develop the migration programs, and QA staff to validate the data. Select staff that have experience working with the data and applications involved.
- Identify data sources and data interdependencies. This can include legacy systems, applications that interface with the data, popular user reports, and desktop spreadsheets and databases. Ensure that business managers and end users drive this task.
- Perform a quick assessment of the data that will be migrated. Count the number of fields and records that need to be migrated to make a rough assessment of how long the migration will take.
- Archive data to reduce the amount of data to migrate. Look at the data migration as an opportunity to de-clutter. If data is no longer being used, archive it. Archiving data will streamline the migration, saving time and potentially reducing the amount of storage space required in the new system.
Tip: For more information on de-cluttering, refer to the McLean Report research brief, “Space: It's Not Endless.”
To keep the new system de-cluttered, develop a document retention policy. Refer to the McLean Report research brief, “Paperless in Steps: Developing a Document Retention Strategy.”
- Create a staging area where programmers can test migration programs.
- Create a subset of the data for testing. For example, copy every 20 th record. The goal is to get a representative sample of data.
- Develop migration rules by analyzing the data to be migrated and activities that might be impacted (refer to the data interdependencies identified earlier). Programmers create a detailed data map of source and target data fields, as well as migration rules. Here are some examples of migration rules:
- Combination of fields (e.g. area code and phone number).
- Separation of fields (e.g. first name and last name).
- Records that will not be migrated automatically (e.g. all customers assigned to sales rep #3).
- Test the migration programs. End users are absolutely essential in this step because they understand the data much better than programmers and can quickly spot errors to be corrected. Involving users also reduces their resistance to change. Validating the results of the migration will make users think of extra rules that must be applied.
Build test scenarios with the following metrics:
- Expected number of records migrated.
- Actual number of records migrated. If the actual is less than the expected number of records, analyze why and refine the migration process.
- Number of fields successfully populated. Randomly sample some records to determine whether every field has been successfully migrated.
- Incorporate changes to migration programs and repeat the testing. Create a larger data subset and repeat the process of analyzing the data, developing scripts, and testing the scripts. Migrating large quantities of data is a highly iterative process.
- Migrate the live data and test the output. Perform the migration during off-hours when the affected systems can be locked down with minimal impact.
- Publish the rules used to migrate the data along with the documentation for the new system. This makes future projects such as maintenance significantly easier.
To be successful, a data migration plan must account for data interdependencies and follow an iterative process that includes analyzing a data subset, developing migration programs, and testing the output before the live data is migrated.
© 1998-2010 Info-Tech Research Group. All rights reserved. Reprinted by permission
Info-Tech's products and services combine actionable insight and relevant advice with ready-to-use tools and templates that cover the full spectrum of IT concerns. For more information, go to www.infotech.com.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access