Four Places Where You'll Find Custom Integration Code -- But Shouldn't

Loraine Lawson

When I asked data-integration vendors who their biggest competitor is, they don't mention IBM or Informatica or any other vendor. Their answer is nearly universally custom code.


It's a well-known fact in the data-integration space that hand coding is often preferred over ETL and other data-integration tools. This causes analysts and vendors no end of headaches, because they say custom code costs more in the long run. The problem is so extensive, you even find custom code chosen in companies that already have an ETL tool and recognize its use as a best practice, according to Rick Sherman a consultant with Athena IT Solutions and the Data Doghouse blogger.


In a Beye-Network article published this week, Sherman says assuming custom code is faster than ETL development is one of the most common mistakes companies make with data integration or extract, transform and load processing. And he only lists three mistakes in total, the other two being thinking of data quality as an end product, not a process, and not designing an overall data-integration architecture.


So, how can you ensure the IT staff is using ETL first and hand-coding only when necessary? You can start by locating it. Sherman lists four places you'll find custom code hiding:

  1. Small and medium-size businesses, which in general are not using ETL tools as extensively as larger companies.
  2. Using a database vendors' ETL tools-particularly older versions-to run SQL scrips or store procedures. Sherman explains that while this is technically using ETL tools, they're actually writing custom code for the integration and then using the tool to run the code.
  3. BI applications that require summary or aggregation tables. Sherman says these tables often are built with SQL scrips or stored procedures.
  4. Business groups, who code to create shadow data systems. Here's a clue that this might be going on: They'll ask IT for a flat file extract, or use a BI tool to create one, then they use Microsoft Access or Excel to do the ETL work.


Sherman doesn't debate how he knows ETL can be faster than hand-coding; he simple says, "Let's just say that when I see a large block of custom code, I see a great opportunity for change and improvement."


But David Linthicum offered a meatier explanation of why ETL is a better choice than hand coding in a post that essentially seconded Sherman's piece:

"Everyone thinks that they can code their way to success here, when in fact it typically means that your ongoing maintenance costs, and also the cost of inefficiencies, are off the charts. In this day and age, unless there are specific and unique needs, custom coding your data-integration solution is never a good idea."


Of course, part of the problem is cultural. Developers like to code. So if you're going to take full use of that ETL tool you've bought, you're going to need to sniff out the places where custom code likes to hide and decide if a tool would work better.


For more on ETL, ceck out these IT Business Edge resources:

More from Our Network
Add Comment      Leave a comment on this blog post

Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.