Legacy Data Migration Planning and Approach

What is Legacy Data Migration and Why is it Needed?

Legacy Data Migration is the process of moving data from one storage system to another. As companies move systems such as from a home-grown system to a Commercial Off-the-Shelf software (COTS) or from one COTS to another, there is a desire to move legacy data from the older application into the new application. Data may have to be moved out of older systems for various reasons such as decommissioning of the old system, to enhance performance, to reduce upkeep of multiple systems, and for application consolidation.

Legacy Data Migration Planning

Legacy Data Migration is often time consuming and expensive. As data becomes more complex, you may need to spend more time preparing, transforming, and cleansing data for import into the new system. Legacy data migration planning includes identifying business rules to help focus your efforts on making sure only the legacy data you need is moved to your new system. These business rules can be factors such as regulatory reporting requirements, company record retention policies, data importance, and Key Performance Indicators (KPIs) trending and analysis over time. The data not being migrated into the new system can still be preserved by storing it in database tables, Excel spreadsheets, or PDF extracts of reports on a network drive or a central location such as SharePoint, in case it needs to be reviewed later or for trending and analysis using Business Intelligence Tools such as Microsoft Power BI or Tableau.

Legacy Data Migration Approach (Extract, Transform, Load)

ALL4 follows the ETL (Extract, Transform, and Load) approach for legacy data migration.

Extract – Once business rules are established for the data to be migrated and what information (fields) must be extracted from the legacy system are determined, mapping is developed between the legacy tables/fields and the destination system tables/fields to ensure everything has a home in the system. Data is usually extracted in Excel spreadsheets or delimited text files to load into the new system.

Transform – Business rules will guide the transformation and cleansing of data. Transformation may be required for multiple reasons, some of which are:

1. The new system could have different options or nomenclature for choice lists than the legacy system.
2. There could be legacy data that does not have a one-to-one match in the new system, but the data has to be preserved for reference.
3. There may be required data in the new system not previously required in the legacy system.
4. The legacy system may have records such as Corrective Actions or Tasks associated with an employee who is no longer with the company, but the information has to be preserved without creating the inactive employee records in the new system.

Once the transformation rules are established, procedures should be developed to apply rules as uniformly as possible. In some cases, it may not be possible to apply the transformation rules programmatically and manual intervention may be required. The end result of this step is fully populated import templates that can easily be loaded into the new system.

The transformation process may include cleanup of data quality issues often encountered with older data.

Load – A small subset of data should first be loaded into a TEST environment to confirm data is being imported correctly and appears in the correct locations in the application. This allows errors to be caught early and corrected for proper loading in the system. After data is successfully loaded into the TEST environment, data can be imported into the production environment. Data checks are done on the imported data by comparing the record counts in the import templates with the uploaded record counts. Thorough testing should be done to make sure all the data is migrated and correctly appears in the application and reports.

Depending on the volume of data and the implementation schedule for the new system, the ETL process may have to be done multiple times. If the legacy system is still online when the initial data set is extracted, a delta extraction, transformation, and load will have to be done either during a blackout period right before the Go Live of the new system or post cutover to the new system.

Summary

Legacy Data Migration is a critical task and can be very involved, but proper preparation, planning, and execution can reduce costs and headaches. A well-thought-out strategy will help your new system be a success and improve adoption of the new system by internal stakeholders.

ALL4’s Digital Solutions Practice has extensive experience helping clients with legacy data migration. Our team can help in developing the strategy and performing the legacy data migration. If you would like to discuss further, please contact Anubha Garg at agarg@all4inc.com or 281-201-1245.