Data Integration Remains a Major IT Headache
Study shows that data integration is still costly and requires a lot of manual coding.
You did it. You heeded all the admonishments to stop hand coding and invested in a data integration platform/solution/tool-whatever. The point is: You listened and now your development team doesn't have to hand code all your integration work.
And now, lo and behold, your developers are complaining that the tool takes too much time and is just as much work as hand coding. Is the tool useless? Did you buy the wrong solution? Or did you just get swindled by hype?
The answer, in all cases, is probably no. The most likely problem lies in your programmers' understanding of the mechanics of data integration, says Rick Sherman of IT Athena Solutions, who writes the Data Doghouse blog.
That's why Sherman says the data integration tool you use doesn't matter as much as understanding the data integration process and its best practices.
Sherman actually takes a pretty relaxed view on choosing a tool. "Almost" all data integration solutions provide the core capabilities needed to follow industry best practices, he writes. But it's hard for integration developers to leverage those tools if they don't understand the mechanics of data integration. In that case, they're likely to revert to what they do know and do the equivalent of manual coding, using the tool as a blunt instrument for importing, extracting and transforming - all of which leaves developers frustrated because they feel they could've hand coded it faster without the tool. If your developers tell you that, you might see it as a red flag: They probably don't understand data integration processes, he warns.
It sounds like an odd problem to have, but Sherman says it's not unusual. A few years ago, I interviewed John Shafer, the application developer for the e-business portion of Levolor. The IT division had recently adopted a data integration platform to replace hand coding. Shafer explained that it caused something of a coding culture shock for him:
It was difficult at first to step back because you think of how you approach a problem if you could code it as opposed to using some components. I come from the background of being able to code it, so you have unlimited ways of getting there. Now, with Talend, you kind of step back and say, 'Well, I've got all these components, what is the best way of getting there that lends itself to clarity - for someone else to clearly look at the job and see how it's set up and see how it's meant to operate?' ...You have to figure out how to leverage those. So it's just a little different mindset that initially is a little difficult, at least it was a little difficult for me.
More companies are probably encountering more of this shock this year than in previous years, since a 2010 TDWI survey revealed that a growing number of companies are shifting away from hand coding to using a data integration tool-the first time the TDWI has ever seen that trend.