Top Ten Best Practices for Data Integration
Use these guidelines to help you achieve more modern, high-value and diverse uses of DI tools and techniques.
Changes are afoot in the data integration market, and no company embodies those shifts more clearly than Talend.
In its recent Magic Quadrant for Data Integration Tools, Gartner identified several trends that are affecting the market, including a growing interest in open source solutions and lower-cost options-both of which apply to Talend. I wrote about some of these trends last month, but there's one trend that I'd like to focus on today that is particularly relevant to your options as a consumer of data integration solutions: the diversification of solutions we're seeing from data integration.
It's no longer all about ETL, and while that's been true for some time, the net is widening, with vendors moving into master data management (MDM) and even solutions traditionally associated with application integration, such as ESBs and a service-oriented focus.
In November, Talend bought into that market by acquiring Sopera, which sells an ESB solution marketed for SOA. I spoke with Talend's CEO and co-founder Bertrand Diard, as well as VP of Marketing Yves de Montcheuil, about this news a few weeks ago, but at that time, they were still working out the details of how this would impact their product offerings.
This week, Talend unveiled its plans for integrating Sopera's technology over the next few years. Not surprisingly, its MDM solution will be integrated with Sopera's application integration capabilities and service-oriented approach. The ESB will also be added to its data integration solution, which will mean support for data services. Meanwhile, it will continued to offer the Sopera Advanced Service Factory under the same brand name, thus retaining Sopera's SOA business. This SOA open source connection prompted CTO Edge's Mike Vizard to ponder whether open source might further drive SOA's adoption.
As a side note, Talend also plans to offer new application integration products for developers using projects from the Apache Software Foundation. What's more, Talend still has a chunk of change from its recent $34 million funding round, and it's looking to spend that money on new acquisitions.
Informatica has also recently moved to the MDM market, acquiring MDM pure-play vendor Siperian early in 2010.
To me, it seems like a natural evolution for data integration vendors like Talend and Informatica to expand, particularly when it comes to data quality-an area of convergence for some time-and MDM. After all, MDM is essentially about integrating data and solidifying on a "golden record" for master data.
But not everyone thinks it's that simple.
Recently, I spoke with MuleSoft founder and CTO Ross Mason and its VP of Engineering Ken Yagen about this very issue. MuleSoft is an open source vendor, specializing in ESBs. Its theory is that ETL is a "very old world way at looking at moving data around the enterprise," as Mason put it:
What we're seeing is the location of data is changing, so it's not only in databases anymore. We're seeing a lot more unstructured data or even data in cloud applications, SaaS applications. I just think the demands on the need to gather information in real time has also changed. ETL used to be very sort of batch-oriented, but what the enterprise really needs is real time information. The ETL model is becoming more obsolete because of demands on how we use the data. So I don't think there is anything wrong with ETL, in terms of what it was supposed to do, but it's like any technology has a shelf life. The world changes and the technology has to change with it.
Yagen added that ETL solutions were designed to address application silos, but with the advent of service-oriented architecture, those silos are breaking down. The data is no longer locked away, which means there's less demand for batch processing to pull information from a back-end database or application.
Of course, this trend isn't just about Talend, but it's certainly an interesting case study for what's happening in the market-and that may be one reason Gartner named it a "visionary" in the recent Magic Quadrant report.