Aberdeen: More Organizations Tackling Complex, External Data

Loraine Lawson
Slide Show

Data Integration Remains a Major IT Headache

Study shows that data integration is still costly and requires a lot of manual coding.

Twenty-five percent of integration costs go to integrating with data outside the organization's walls, according to "Future Integration Needs: Embracing Complex Data," a report by the Aberdeen Group and Informatica. Is anybody else surprised at how low that number is?


CBR Systems and Network Storage points out this means "Integration of external data continues to be a labor- and cost-intensive task," but if 25 percent is dedicated to external data, then doesn't that mean the rest is spent on integrating internal data?


Further, if you're spending most of your time just trying to integrate information within your own systems, rather than bringing in new data for consumption, well, I can't help but wonder if an issue involving the tactical-versus-strategic use of integration is underlying this statistic.


I downloaded the report-which is available for free from Informatica's site-and discovered there is a correlation between those who focus on integrating external data and strategic goals like performance gains. For example, those who integrate externally with key retailers can deliver perfect orders to customers 78 percent of the time; those without that integration can deliver perfect orders only 69 percent of the time. That's significant at a nearly 10-percentage-point difference.


Another interesting finding in the report: The rest of that budget is spent on software licenses, services and support, and internal support. The bulk of the funds go to internal support at a hefty $555,000 a year. Ouch.


You won't be surprised to learn that organizations are managing more data these days, but you might be shocked by how much more. The average data volume grew by 40 percent last year, according to Aberdeen. And more of it is coming from external sources, with organizations integrating an average of 14 external data sources such as third-party partners and online data services.


Also, that data is more complex. We're talking unstructured data from office documents, email, Web content and social media here.


These shifts probably explain the boost in XML adoption, with 74 percent integrating external XML data sources and 66 percent integrating XML from internal sources.


That translates into a shift in integration work as well. Seventy-four percent of organizations said they were integrating XML from external sources, compared to 66 percent integrating XML from internal sources-which, again, makes you wonder where that other 75 percent of integration spending is going. Maybe it'll be reflected in next year's integration budget spending?


But here's what's interesting: Once you can integrate all this complex data, you get better at a lot of things. The report found that those organizations that can manage complex data also:


  • Are twice as successful at integrating external, unstructured data into business processes
  • Use up to 50 percent larger data sets for business intelligence and analytics
  • Deliver "critical information in the required time window 2.5 times more often via automated data refresh"

So, in other words, once you can handle complex, unstructured data, well, you can handle larger data, which could lead to better BI and analytics, and you can do it faster. But that's not all-those companies are better at data quality, cutting the "the incidence of errors in their data almost in half compared to organisations relying on manual intervention." And they do all this while spending less-an average of 43 percent less-on integration software.


To prepare for this shift, Aberdeen suggests companies focus on integrating with data souces where there's quality assurance, automating the data integration and the transformation of complex data, and consolidating onto a single data integration platform.

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


Add Comment      Leave a comment on this blog post
Sep 14, 2011 12:00 PM Yves de Montcheuil Yves de Montcheuil  says:

Loraine - thanks for this great summary. I think there is a good reason companies only spend 25% if their budget on integrating external sources: they is still a lot of work to do with internal sources!

Many vendors want us to believe that integrating Cloud and SaaS is the only way to go. And it is the way to go, BUT - Cloud and SaaS systems need to be integrated with the rest of the information system. This is what is called an hybrid environment, and this is our key focus in our Talend Cloud offering.

At the end of the day, the key is to integrate holistically all your systems - Cloud, SaaS or on premises. For this, you need a solution with the right set of connectors, with the right hybrid deployment options, and with the right price & flexibility, that does not leave systems behind because the license costs are too great...

Yves de Montcheuil


Sep 23, 2011 4:33 PM Ben Rose Ben Rose  says: in response to Yves de Montcheuil

Yves makes a good point, and yes, many companies are still struggling with organizing internal data to serve different groups.  I also think it is important to remember that companies are not getting external data for it's own sake.  The point is to see how that external data relates to events within the system and what actionable insights can be gained.  This usually means additional effort spent on integrating internal data for yet another purpose.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.

Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.