Data Integration Remains a Major IT Headache
Study shows that data integration is still costly and requires a lot of manual coding.
Due largely to performance issues and the sheer volume of data involved, the whole extract, transform and load (ETL) process has been gaining an increasingly bad reputation within IT circles. The core issue is that current approaches to automating ETL simply don't scale, while the amount of time that IT organizations can allocate to support this kind of batch processing is shrinking.
As a result, the folks at Syncsort are starting to make a case of ETL 2.0, which in their view will soon lead ETL engines running in memory as a way to solve the current performance and scalability limitations of ETL.
In a Big Data era, the whole data integration process is lagging behind recent advancements in memory technologies. Jorge Lopez, Syncsort senior manager for data integration, says the sheer complexity of all the data sources involved in a modern application integration project are starting to overwhelm existing approaches to ETL.
Lopez says it will take a little longer before Syncsort can move its ETL engine into memory, but does say that IT organizations should start planning today for what he says will happen sooner than later. That means ETL technologies are something that IT organizations need to work around, and the next generation of ETL platforms will be technologies that IT organizations will find hard to keep up with.
At a time when data integration across multiple platforms residing both inside and outside the enterprise is increasingly becoming critical, advances in the way the whole ETL process is performed and managed can come none to soon.