Signs Your ETL Tool Is Slowing You Down

Loraine Lawson
Slide Show

Top Ten Best Practices for Data Integration

Use these guidelines to help you achieve more modern, high-value and diverse uses of DI tools and techniques.

Claudia Imhoff, founder and president of Intelligent Solutions, Inc., sees data integration as the bane of data warehouse and BI workers' existence. And given her estimate that fooling with ETL takes up anywhere from 60 to 80 percent of the project lifecycle effort, you can see her point.

 

That's a significant time suck, so any problems in ETL performance are worth hunting down and eliminating, said Imhoff in an on-demand TDWI webinar. You will need to register to view the presentation.

 

Last week, I shared David Linthicum's three tips for giving your data integration process a performance tuneup. Imhoff goes into more depth about how to tell if you have an ETL performance problem and looks at more solutions companies typically use.

 


Her checklist of red flags indicating ETL problems is actually quite useful, since, as Imhoff notes, most data integration performance issues tend to slip up on you. They become chronic problems that are easily overlooked. The next thing you know, you've bought more hardware, shifted some of your ETL processing to the data warehouse and still data integration is sluggish at best. That's because you fixed the wrong problem.

 

Her ETL red flag list includes:

  • Buying more hardware, but only seeing a modest increase in performance
  • Frequent need to hand-code a "user exit" or offload processing to the database to work around the ETL
  • Difficulty with tracking data and transformation flows through the entire process
  • Maintenance takes longer and uses more resources
  • Data integration projects and components take longer to deliver

 

She also discusses-and largely rejects-the most common attempts to solve data integration problems. For instance, many companies opt to add more hardware, and while that can work, it's an expensive workaround that may run into trouble if your software isn't scalable. It also ignores the underlying problem.

 

Instead, she suggests you hunt down bottlenecks and focus on using technology to address the specific problem.

 

I know. It sounds obvious. But apparently, that's not what people typically do.

 

The presentation also includes a discussion by Syncsort's senior manager for product marketing of data integration.

 

Imhoff plans to write a downloadable checklist for TDWI with more information, although she didn't say whether or not it would be available for free.

 

While you're on TDWI's site, you might want to register for some of the upcoming events, which include an April 12 presentation by TDWI Research Director Philip Russom on "Next Generation Data Integration," and an April 27 webinar, "Benchmarking Data Governance," featuring Andy Hayler of Information Difference.



Add Comment      Leave a comment on this blog post

Post a comment

 

 

 

 


(Maximum characters: 1200). You have 1200 characters left.

 

null
null

 

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.