Data Integration Remains a Major IT Headache
Study shows that data integration is still costly and requires a lot of manual coding.
Twenty-five percent of integration costs go to integrating with data outside the organization's walls, according to "Future Integration Needs: Embracing Complex Data," a report by the Aberdeen Group and Informatica. Is anybody else surprised at how low that number is?
CBR Systems and Network Storage points out this means "Integration of external data continues to be a labor- and cost-intensive task," but if 25 percent is dedicated to external data, then doesn't that mean the rest is spent on integrating internal data?
Further, if you're spending most of your time just trying to integrate information within your own systems, rather than bringing in new data for consumption, well, I can't help but wonder if an issue involving the tactical-versus-strategic use of integration is underlying this statistic.
Another interesting finding in the report: The rest of that budget is spent on software licenses, services and support, and internal support. The bulk of the funds go to internal support at a hefty $555,000 a year. Ouch.
You won't be surprised to learn that organizations are managing more data these days, but you might be shocked by how much more. The average data volume grew by 40 percent last year, according to Aberdeen. And more of it is coming from external sources, with organizations integrating an average of 14 external data sources such as third-party partners and online data services.
Also, that data is more complex. We're talking unstructured data from office documents, email, Web content and social media here.
These shifts probably explain the boost in XML adoption, with 74 percent integrating external XML data sources and 66 percent integrating XML from internal sources.
That translates into a shift in integration work as well. Seventy-four percent of organizations said they were integrating XML from external sources, compared to 66 percent integrating XML from internal sources-which, again, makes you wonder where that other 75 percent of integration spending is going. Maybe it'll be reflected in next year's integration budget spending?
But here's what's interesting: Once you can integrate all this complex data, you get better at a lot of things. The report found that those organizations that can manage complex data also:
So, in other words, once you can handle complex, unstructured data, well, you can handle larger data, which could lead to better BI and analytics, and you can do it faster. But that's not all-those companies are better at data quality, cutting the "the incidence of errors in their data almost in half compared to organisations relying on manual intervention." And they do all this while spending less-an average of 43 percent less-on integration software.
To prepare for this shift, Aberdeen suggests companies focus on integrating with data souces where there's quality assurance, automating the data integration and the transformation of complex data, and consolidating onto a single data integration platform.