Can Switching the T & L in ETL Save You Money? Maybe

Loraine Lawson

Does it really matter when you load and when you transform data during a data integration project? Yes, actually it does, argues a recent Intelligent Enterprise article, "Is it Time to Switch to ELT?"

 

The article, written by Sreedhar Kajeepeta, compares the traditional extract-transform-load approach to data integration with an extract-load-transform approach-and a few hybrids that extra "transforms" at various points in the process.

 

I know it sounds a bit overly technical and perhaps even like nerdy nit-picking, but there's actually a strong cost and efficiency issue here, particularly in this age of virtualization and cloud computing. Loading before you transform actually gives you cost-of-performance advantages, particularly in data warehouse and business intelligence initiatives by reducing the costs of software licensing and development.

 

By contract, transforming before you load saves in infrastructure costs, but that's becoming less of an issue, thanks to cheaper multicore database servers, appliances, virtualization and cloud computing. That's why you'll be hearing more about ELT, Kajeepeta says:

As an enabler of this bigger BI market, the field of data integration tools is expected to grow 17 percent annually to reach $3 billion by 2012, according to Gartner estimates. It is in this market that we are likely to see ETL and ELT battling it out for market dominance. With performance and low latency increasingly in demand, you can guess ELT will get a lot of attention.

Kajeepeta offers clear guidelines for when you should stick with the old ETL and when you should consider an ELT or hybrid option:

As a rule of thumb, it's generally accepted that you should stick with ETL for DW/BI projects when there are 10 or more source systems or when the source databases are a terabyte or larger in size. If your project doesn't fit this description, it's time to consider ELT and to match tools against your requirements.

He also offers a few other issues to consider:

  1. ETL requires investments outside the database management system; ELT assumes the availability of strong DBA skills.
  2. ETL is more evolved in terms of connectivity so, for instance, if you're dealing with GIS or another unusual data source, ETL is probably the best option.
  3. ELT uses the hardware infrastructure for linear scalability, which means your relational database management system must be able to scale for very large databases, he writes.
  4. Vendor lock-in is more likely with ELT.

 

I'll be honest. I'm not sure how much credence to give this issue. I noticed it surfaced as a discussion point in 2006 - Vincent McBurney of IT Toolbox had an good piece explaining the pros and cons - and then disappeared until 2008, when it made a brief reappearance, and now it's back. Usually, that's a sure sign of a vendor trying to push a sales angle as a bigger story.

 

But there are two things that make me think this is worth considering and not just a vendor pitch. First, Kajeepeta is the the global vice president/CTO of technology consulting practices for the Global Business Solutions & Services division of CSC, an India-based services firm, not a data integration company. Second, Kajeepeta discusses how several vendors approach this issue, including Oracle (ELT), IBM and Informatica (both multiple options), and appliance vendors Netezza and Teradata, (ELT).

 

So, I think it's worth considering. It's also surprisingly understandable, if you can keep the acronyms straight.



Add Comment      Leave a comment on this blog post
Jun 9, 2010 2:34 AM Yves de Montcheuil Yves de Montcheuil  says:

Loraine, when I was at Sunopsis we more or less invented ELT and certainly coined the term. However Sunopsis - now Oracle's ODI - is an ELT-only solution.  Other vendors such as Informatica or IBM support primarily ETL and have tried to instill some ELT in their products for fear of loosing the buzz war, but it's only a half hearted attempt, because their business model consists of selling more CPUs of the transformation engine, and ELT kills this model. 

What I don't understand is why Sreedhar Kajeepeta only mentions a handful of legacy tools, when some more agile (and more open) tools like Talend support natively ETL and ELT in the same process, allowing you to pick and choose what works best for each individual transformation.  And I won't even get into the openness and cost elements...

Reply
Jun 9, 2010 10:57 AM Loraine Lawson Loraine Lawson  says: in response to Yves de Montcheuil

I don't know either. It was a long piece. The information could've been cut, I suppose, with the author or editors deciding to give a few samples rather than a comprehensive look at the market.

Reply

Post a comment

 

 

 

 


(Maximum characters: 1200). You have 1200 characters left.

 

null
null

 

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.