No More Weekend Data Migrations: Another Reason for an Integration Competency Center

The weekend migration is practically a rite of passage for IT, but it may soon become only a legend old timers tell the newbies.

A recent post on Data Migration Pro points to two trends that could make big-bang data migrations history:

1. The growth of data
2. The use of complex, but slow, APIs, which slow down data movement

“So, you have a situation where your data can be extracted with an ETL solution (or equivalent) at insanely high speeds through your export, transformation and cleansing stages only to screech to a halt as it meets an API that loads at sickeningly slow speeds,” writes Dylan Jones, the founder of the site, in a recent post.

It’s already become a problem for telecom migrations, he continues. In some cases, the amount of data, limited by the API’s cap on movement, could mean you’re moving data for a week rather than a weekend. Obviously, that’s a problem.

So what can you do instead? He discusses a few options.

The first, and I’ve seen this recommendation time and time again from data experts: Decouple the data from the applications. SOA is a great tool for this, as he points out, but it’s going to take a significant shift in development shops and some time to resolve this piece of the puzzle.

The second solution is iterative data migrations, which present their own challenges: Keeping the target and source data in sync during the migration and managing the business end of things while you do the migration.

Obviously, these are not insignificant problems, but they are manageable, as he explains in the post.

This shift won’t just be the end of an IT ritual — it will also require mean data migrations that will require more involvement and cooperation with the business. I can’t help but think this is yet another argument for establishing an Integration Competency Center, or, at the very least, sitting down for a serious conversation about best practices and your own experiences with migrating or integrating data.