Sometimes, writing about data integration feels a little like watching Charlie Brown try to kick the football while Lucy’s holding it. It looks like the same problems keep happening over and over again: shadow IT, data silos, rogue spreadsheets.
If you feel the same way, check out this post by David Linthicum, senior vice president of Cloud Technology Partners and veteran data integration, SOA and cloud writer.
Linthicum provides much-needed perspective by referencing an Information Management article on data integration — from eight years ago. At that time, 69 percent of companies told TDWI that data integration issues were a high to very high barrier to new application development.
They planned to spend more on data integration products. Of course, about five to six years later, we’d be talking about the cloud, SaaS and then Big Data.
“The issue I had at the time was the inability to deal with real-time operational data, and the cost of the technology and deployments,” shares Linthicum. “While these issues were never resolved with traditional BI and data warehousing technology, we now have access to databases that can manage over a petabyte of data, and the ability to cull through the data in seconds.”
In that time, data integration has also changed, he adds, particularly since the introduction of Big Data tools into the enterprise. That’s made data integration a more strategic issue for many companies.
“Millions of dollars an hour of value are being delivered to Global 2000 organizations that leverage these emerging data integration approaches and technology,” he writes.
Of course, Big Data and cloud integration have also introduced new challenges. That’s why data integration should top your to-do list when evaluating cloud-based Big Data analytics, advises Linthicum in a recent TechTarget article about Amazon’s Web Services tool.
“Data integration is the first problem you need to consider when doing big data analytics in the public cloud, whether it's with AWS or another provider,” he states. “Your data needs to flow from operational data stores in your organization to your big data systems, most likely in the cloud.”
If you’re interested in AWS as a solution, Linthicum identifies several solutions that can help you with handling integration with the platform.