When I asked data-integration vendors who their biggest competitor is, they don't mention IBM or Informatica or any other vendor. Their answer is nearly universally custom code.
It's a well-known fact in the data-integration space that hand coding is often preferred over ETL and other data-integration tools. This causes analysts and vendors no end of headaches, because they say custom code costs more in the long run. The problem is so extensive, you even find custom code chosen in companies that already have an ETL tool and recognize its use as a best practice, according to Rick Sherman a consultant with Athena IT Solutions and the Data Doghouse blogger.
In a Beye-Network article published this week, Sherman says assuming custom code is faster than ETL development is one of the most common mistakes companies make with data integration or extract, transform and load processing. And he only lists three mistakes in total, the other two being thinking of data quality as an end product, not a process, and not designing an overall data-integration architecture.
So, how can you ensure the IT staff is using ETL first and hand-coding only when necessary? You can start by locating it. Sherman lists four places you'll find custom code hiding:
Sherman doesn't debate how he knows ETL can be faster than hand-coding; he simple says, "Let's just say that when I see a large block of custom code, I see a great opportunity for change and improvement."
But David Linthicum offered a meatier explanation of why ETL is a better choice than hand coding in a post that essentially seconded Sherman's piece:
"Everyone thinks that they can code their way to success here, when in fact it typically means that your ongoing maintenance costs, and also the cost of inefficiencies, are off the charts. In this day and age, unless there are specific and unique needs, custom coding your data-integration solution is never a good idea."
Of course, part of the problem is cultural. Developers like to code. So if you're going to take full use of that ETL tool you've bought, you're going to need to sniff out the places where custom code likes to hide and decide if a tool would work better.
For more on ETL, ceck out these IT Business Edge resources: