Loraine Lawson spoke with Evan Levy, a partner at Baseline Consulting and an instructor at The Data Warehousing Institute, about why data warehouses aren't the place to start multipurpose integration projects, aka master data management, and why application developers tend to “get” MDM more than the data warehousing developers.

Lawson: You taught a TDWI class called Architectural Options for Data Integration. In a recent blog post, you said people often come to the class intending to justify a data warehouse as a multipurpose integration system. Can you explain what’s wrong with that idea?
Levy: Data warehousing first came into place because people needed to run reports and they couldn’t get the data off their operational system or the data was in two or three places, so they put it onto the data warehouse. And they quickly learned that there’s a fair amount of effort in trying to integrate the values themselves.

One of the things that you find now is that data integration is probably the single largest development or coding cost inside an IT organization. By and large, very few IT organizations do any sort of coding anymore. I mean, yeah, they all claim they do a lot of development, but by and large it’s a lot of package implementation or configuration with one exception: The data integration area.

One thing that I’ve seen, particularly when these data warehouses get large -- and they all grow, and they don’t grow linearly, they tend to grow exponentially as people find value -- is the single area that has integration skills is the data warehouse. Unlike an SAP environment or a financial package or a CRM system that typically only retrieves data from other systems at a single subject area level, the data warehouse tends to have multiple subject areas; it’s cross functional. So the level of knowledge about how to link customer IDs with orders, orders with products, products with suppliers and so forth - those skills probably only exist in a lot of companies in the data warehouse group.

So lo and behold, as things like master data management or even ESBs have come into play, people are trying to squeeze more value out of their warehousing environment so they're trying to say, “You know what, let all the data be integrated on the data warehouse platform. Don’t do it anywhere else.”

That’s a very naive view because of the inevitable latency and complexity that putting something onto a data warehouse creates. There’s also this presumption that to integrate data I have to pick it up, copy it into one place to integrate it, and in fact MDM challenges that paradigm completely. It says leave it where it is and keep track of it and retrieve and create the records when they're necessary.

Previous Page Next Page