17 Sep 7 steps to successful data migration – how to migrate to the cloud
Many organisations are now considering a move to the cloud from legacy on-premise solutions and systems. There are several considerations as you review each application for migration. Underlying each application migration project is a data migration exercise that moves all your associated data from your servers in-house into the cloud service providers. Data migrations can be one of the most complex aspects of the move to the cloud and to get it right requires careful planning. Our Head of Platforms Craig Broadbent outlines 7 key steps in this process and will help make your cloud migration successful.
1-Analyse the data
You really need to “know your data” before you look at any migration. Understanding what data is where and why (application data, reporting data, etc) is vital in ensuring your business will continue to run as it does today post the migration. Some organisations know their data a lot better than others – and this can be because they don’t have a lot of volume, or because they have put in the effort to understand and catalogue the data in the organisation. Tools such as Alex Solutions and Microsoft Purvue, alongside other data cataloguing solutions, can help in this regard. Understanding how each data asset is used and its lineage, or how it is connected to each application that uses it, will help ensure the migration is successful and will also allow you to determine which data assets can be decommissioned as part of the migration process.
Once the data has been analysed you can also review the quality of the data and determine what sort of clean-up process may be required prior to or during the migration. No matter how well you manage your data you can be sure you will have issues with quality as very few organisations have business wide data reviews to ensure data quality remains at a certain standard, even if those standards are well defined and documented.
2-Plan and Design the migration process
Planning will help ensure business continuity during the process. If you have large amounts of data, you will most likely want to “chunk” the migration process down, as moving large amounts of data will take time. Your previous step will have informed which chunks of data must move together and this will be important input into the planning process—ensuring all business applications work as expected post-migration requires careful planning and understanding of each of your applications and the data used.
If you are going for a “big bang” migration rather than chunking your data down, then testing timings will be important to ensure you have enough scheduled downtime (assuming your applications need down time – most will) to transfer the data to the new location. This can raise pressure on the process as the business will probably need to operate without one or more of their applications online during the process. Test runs on a development server will help. Still, these test runs don’t always give accurate timings as production servers, and production data can often be very different to what happens is stored in our test and development systems.
Whether you decide to migrate all your data together or small amounts at a time (the latter is considered the least risky), careful planning commensurate with the size of your organisation and the amount of data to be migrated is highly recommended. What is also recommended is that you very carefully analyse your data before your start, so that you what stat it is in (from a quality perspective) and exactly where it all is. For example, I once worked with a client where all their CSM’s were storing the phone number in the 2nd address line because their systems were so old they didn’t allow for the storage of a mobile number.
3-Choose the right tool to assist and make sure the people are in place to execute the data migration process is essential. An ETL process where you will extract data from your existing data store, transform it (if required) to suit the new environment, and then load it into the target.
A number of the cloud vendors have tools that you can use to assist with the migration process, or open source solutions can help. Ensuring that whatever tool you choose can do the job by thoroughly testing or asking an independent third party will help ensure your migration happens smoothly and as efficiently as possible.
4-Make sure backups are in place and current
Data migrations, like any complex technical process, can go wrong. There is added risk with data migrations. They often need to run over many days or even months. I remember doing a SharePoint migration for a large enterprise where the migration was run over many months—this added complexity to the whole process and careful consideration of the business impact, which was significant.
5-Test, Test, Test
Oh, but it’s only a “lift and shift,” they said. These are famous last words from those who don’t always understand the complexity of moving significant amounts of data from one system (often on-premise) to another (usually in the cloud) – particularly if there is any reformatting of that data along the way.
Testing with small chunks is critical. It’s ideal to test with real data if possible as you will get a good idea of whether or not there will be issues with the quality of the data being moved. If our transformation programs all assume a certain value in a certain field and then we find that all our front-line staff have been using that field for something completely different, then it can cause issues with the migration with either timing or worse, with a process that fails partway through.
6-Execute the plan
Are the people in place? Check
Are the migration tools in place? Check
Are the migration processes in place? Check
Backups are done and current. Check
Source systems and target systems are all available and sized accordingly? Check
Business has been informed and know exactly how long they will be without their systems and what they will need to do when those systems come back online. Checkü
Post migration audit procedures are in place? Check
Ok, let’s go (note the above is not necessarily a comprehensive checklist for your go/no go decision!).
Time to begin the migration. Because timing of data migrations is so important, the initial hours should be monitored closely to ensure the migration is progressing as expected. If things are going much more slowly than anticipated, it may be that one or more of the assumptions made, or the tools being used, is not performing as expected – and the 2-day or 2-week window you have made available may not be enough time. You can usually tell this within hours and if necessary, it is better to call the migration off and understand why it was not performing as expected than to get 80% of the way there and have the business systems unavailable for an additional week – with the business now suffering and pressure increasing on the technical team. This is often where mistakes can happen, or worse still, the business can be adversely impacted by the unavailability of a key or critical system.
7-Follow up and ensure data quality
Even with proper testing there may still be some issues with the data or the systems you are migrating.
Having your application experts and super users from the business available to test all the systems before they are made available to the rest of the organisation will ensure small issues don’t become larger ones. By picking up any “niggles” with the process, they can be addressed before all users are allowed back in to use the applications. The data migration process can be long, and it can be complex, but following the above process and being thorough should mean a successful migration to your new environment.
If done right, it should also mean cleaner, better-quality data and easier to use applications, both from a user perspective and a technical perspective. As migrations to the cloud increase, refining these processes and making them repeatable becomes more important. Sometimes it is best to have someone help define those processes with you upfront – someone that’s has been through this process before and knows the pitfalls. If you don’t have those people in-house, it can be worthwhile hiring a consultancy to assist. It will be money well spent when your first migration goes as smoothly as your final migration.