Maybe It's Time to Give Up Hand-Coding Data Integration

Loraine Lawson

A recent study by IBM user group SHARE shows at least 50 percent of companies still use hand-coded scripts to move data from mainframes to other databases or platforms. The problem with this type of manual integration is the scrips are hard to maintain -- especially when programmers leave the organization, the study found.


This is no small problem, given that organizations with 10,000 or more employees report that anywhere from 51 percent to 75 percent of their organization's data is managed and stored on mainframes. Given that many legacy programmers are mere years from retirement, this issue is becoming the big elephant in the server room, don't you think?


I must say, I wasn't surprised to see that hand-coding stat. That very issue came up recently when I interviewed Philip Russom, an analyst with The Data Warehousing Institute, about data integration trends for this year.


We were sidetracked when Russom mentioned that data integration solutions are very mature -- 15 years of development will do that for a technology -- but then quickly added that "roughly half of data integration solutions are built from scratch with hand coding."


This surprised me, so I asked, "Why are they hand-coding if you can buy a solution?"


It turns out, Russom's been grappling with that question for some time.


In Russom's words, hand-coded data integration is a "fairly old and arcane practice." His case is simple: Data integration tools really came into their own about 2000, right after the big Y2K push. Data integration requires a senior-level programmer typically, he said, which means you'll be paying at least one programmer six figures to spend months coding data integration from scratch.


By comparison, you could build a comparable solution in two to three months with a vendor tool for less money, he said.


I dryly asked if there was a trend for people to figure that out.


Russom stammered a bit, saying:

"Oh boy, well you've touched a nerve here. You know, I've just gotten tired of explaining this to people for about the last eight years now."

Whenever he explains his case, people tell him: Wow, I get the economics. Wow, thank you for enlightening me. BUT we still have this culture of hand-coding.


And, if I may be so bold, apparently they also have a culture of hard-headedness.


So I can see why many hope service-oriented architecture will be able to shake things up a bit. According to the study, about 23 percent of respondents reported their company is involved with an SOA project, with an additional third in the planning or review stages of SOA.


Certainly, vendors offering SOA solutions are looking at legacy systems. Software vendors are releasing adapters based on the Web Services Interoperability and SOAP Simple Object Access Protocol, according to a recent CIO Today article, "Legacy Data Systems: Forklift or Facelift?"


The article is a great resource for learning how old systems can be given new life through Web tools and SOA, as well as when you should simply pull the plug on legacy systems.


David Linthicum wrote recently


SOA is really about a systemic change in the way we do IT, and not really a 'special project.'

If you work in an IT divisions that's done things the same way for 40 years, SOA is your opportunity to revisit not only how the business works, but how your IT organization functions. Take a clue from Russom and start by evaluating how your programmers are spending their time.

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


More from Our Network
Add Comment      Leave a comment on this blog post
Jan 23, 2008 4:28 PM Bill Miller Bill Miller  says:
Thanks Loraine for this post. Moving away from coding data integration would save IT projects a lot of time and money. I having been using numbers more like 75% to 85% of all data integration is still hand coded, versus the "half" that Philip Russom used in your interview. Most analysts and CIOs I talk with anecdotally corroborate my higher range. My company, XAware.org, has launched an open source project aimed at providing a flexible and cost effective alternative to hand coding operational data integration from scratch, taking a web services-base approach. We hope this open source project helps the industry make the turn away from custom coding. Bill Miller, www.XAware.org Reply
Jan 25, 2008 4:30 PM Stephen Pace Stephen Pace  says:
I think you are spot on with this article, Loraine, although I think the tide is slowly shifting. For example, where a company might have hand-built an operational system in the past, they are now looking to ERPs. Similarly, hand coded ETL and BI solutions are being replaced with packaged ETL and BI tools. As you say, the 'middle' part of the solution--the data warehouse--is still quite dominated by custom built solutions that can struggle to keep up with change. However, just as there has been an evolution in the related spaces, solutions are arriving--like Kalido--that can now provide a viable 'buy' option for the buy vs.. build discussion in the data warehouse space that up to now has been the domain of corporate development. Stephen Pace, www.kalido.com. Reply
Feb 28, 2008 9:45 AM John McMahon John McMahon  says:
Loraine, you have identified a real problem here.Having written my share of adapters and ETL I can say that at least in my case there is a tendency to see these projects as somewhat "easy" programming. And the low visibility of this backend work tends to lure developers into thinking that these will be quick and dirty low risk projects.Developers that would never hand-code a user interface find themselves mired in iterating huge datasets as they test and debug every unhandled field length or character set conversion.This type of project is prone to silent data errors and scalability issues. Then there is a typically underestimated cost of testing and deploying ETL/DW/EII projects.And what about re-use? Because it has low visibility and is often considered "scripting" this type of code tends to be the worst of the worst when it comes to object-oriented, well documented code.Our opinion is that developers have better things to do with their valuable time then writing adapters and scripting ETL tasks.At Extentech, our integration products are purpose-built to leave these inefficiencies behind -- and our customers couldn't be happier.There's no need to code for EII when a code-free integration platform like Exteria can automate complex integration tasks without a single line of code.Until they add more hours to a day, you owe it to yourself to check out the free downloads and see the kind of time and money savings a graphical, code-free EII platform can provide.Not to mention any data flow you create in Exteria can be exposed as a secure SOA web service in about 1 minute.http://www.extentech.com/estore/product_detail.jsp?product_group_id=227 Reply

Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.

Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.