Planners are always telling us to think through the possible scenarios and consequences before making a decision. But in my experience, it's not a lack of effort that leads to the problems, it's the scenarios and consequences you didn't even know could happen that create the real difficulties.
As I convolutedly said to my husband last night after a misstep for precisely that reason, the big problems are caused by the things you don't know that you don't know.
As it turns out, this applies to data integration and ETL tools, as well. In a recent post about misuse and under-use of ETL tools, Rick Sherman explains that this is precisely what happens in small IT departments: They don't understand ETL processes, but they don't realize that's the problem. This may be why some IT teams feel "misinformed" about ETL tools. So, they do what anyone would do: They blame the tool:
Without understanding the why of ETL processing, IT developers either quickly become disillusioned with ETL tools or simply under utilize them. Typically these ETL implementations merely result in the ETL tools executing SQL scripts or stored procedures, e.g. Hand-coding.
That's a problem, Sherman explains, for many reasons, not the least of which is it's a waste of time. Hand coding integration is seldom as efficient, effective or as adept at handling errors as ETL tools, he contends. What's more, developers seldom document these processes and if they do, they seldom maintain the documentation, which leads to problems when you have to hire a new developer to support the code.
And it's a growing problem, because, as Sherman writes in an earlier post, ETL tools have become more affordable. Despite the fact you'll still find custom code in places you shouldn't, the lower cost has made ETL tools more accessible to small IT shops. That means more organizations are at least trying to move away from the great time waste that is hand coding ETL.
The solution, says Sherman, is to train your IT staff on the tools AND ensure your staff does understand the processes, including dimensional modeling, change data capture and error handling, among others. He provides a link to his corporate library, which includes a host of free resources on data integration and ETL tools.
But you can only address the problem if you and your staff know it exists:
You don't know what you don't know. It's not that the IT staff wants to use these ETL tools either incorrectly or inappropriately, but they don't know any better.
Right, that's exactly what I'm saying: It's the things you don't know that you don't know that will get you every time.