Automating Integration: Are Your Programmers Doing It Wrong?

Loraine Lawson
Slide Show

Data Integration Remains a Major IT Headache

Study shows that data integration is still costly and requires a lot of manual coding.

You did it. You heeded all the admonishments to stop hand coding and invested in a data integration platform/solution/tool-whatever. The point is: You listened and now your development team doesn't have to hand code all your integration work.


And now, lo and behold, your developers are complaining that the tool takes too much time and is just as much work as hand coding. Is the tool useless? Did you buy the wrong solution? Or did you just get swindled by hype?


The answer, in all cases, is probably no. The most likely problem lies in your programmers' understanding of the mechanics of data integration, says Rick Sherman of IT Athena Solutions, who writes the Data Doghouse blog.


"Most data integration architects, designers and developers started ETL by writing SQL scripts or manually coding using something like Java with JDBC. Then they try to replicate what they did in the manual code into the data integration processes," writes Sherman in a recent post. "This is probably the worst way to use a data integration product! You likely get little benefit from the framework, processing is not optimized (maybe even terrible) and worse, the developer gets frustrated because he feels he could have coded it faster."


That's why Sherman says the data integration tool you use doesn't matter as much as understanding the data integration process and its best practices.


Sherman actually takes a pretty relaxed view on choosing a tool. "Almost" all data integration solutions provide the core capabilities needed to follow industry best practices, he writes. But it's hard for integration developers to leverage those tools if they don't understand the mechanics of data integration. In that case, they're likely to revert to what they do know and do the equivalent of manual coding, using the tool as a blunt instrument for importing, extracting and transforming - all of which leaves developers frustrated because they feel they could've hand coded it faster without the tool. If your developers tell you that, you might see it as a red flag: They probably don't understand data integration processes, he warns.


It sounds like an odd problem to have, but Sherman says it's not unusual. A few years ago, I interviewed John Shafer, the application developer for the e-business portion of Levolor. The IT division had recently adopted a data integration platform to replace hand coding. Shafer explained that it caused something of a coding culture shock for him:

It was difficult at first to step back because you think of how you approach a problem if you could code it as opposed to using some components. I come from the background of being able to code it, so you have unlimited ways of getting there. Now, with Talend, you kind of step back and say, 'Well, I've got all these components, what is the best way of getting there that lends itself to clarity - for someone else to clearly look at the job and see how it's set up and see how it's meant to operate?' ...You have to figure out how to leverage those. So it's just a little different mindset that initially is a little difficult, at least it was a little difficult for me.

More companies are probably encountering more of this shock this year than in previous years, since a 2010 TDWI survey revealed that a growing number of companies are shifting away from hand coding to using a data integration tool-the first time the TDWI has ever seen that trend.

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


Add Comment      Leave a comment on this blog post
Sep 26, 2011 4:30 PM Michael Waclawiczek Michael Waclawiczek  says:


Great blog post.  I do agree with Rick on the notion that most ETL tools are a better solution than writing SQL scripts.  But when it boils down to choosing the right data integration tool, there aren't that many tools that are easy to use, powerful and affordable.


Sep 27, 2011 11:25 AM Ross Ross  says:

Hi, Loraine!

When I was learning how to use Talend, I found that things did take longer at first.  It does require a different way of looking at things.

However, I think there's real value in using data integration tools when dealing with changing data structures and business requirements.  I found it much easier to update metadata for a schema change and propagate it across my entire data flow than it would have been to do the same thing in code.


Sep 27, 2011 5:41 PM Colin B. Colin B.  says:

Great article.

I was curious how you saw automation, specifically tools such as job scheduling and workload automation fitting into the mix of integration, automation and reducing the reliance on custom coding?

Oct 6, 2011 6:42 PM Juhan Leemet Juhan Leemet  says:

Similarly, I have seen many organizations buy a methodolgy tool, try it (half heartedly?) on 1 or 2 projects, and abandon it, going back to "the old way".

I think one needs at least 1 person who truly understands the tool(s), methodology(ies), and applications, in depth. Then that person can "champion" the "new way" by encouraging others and helping them out when they get stuck. Sometimes it's not so obvious how to change one's thinking.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.

Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.