Integration’s Coming of Age Story

So many 50-cent words surround integration that it’s easy to forget that it mostly boils down to two approaches:

• ETL
• Data virtualization

In a recent Inside Analysis post, Barry Devlin swipes away all the hype to explain data integration’s simple coming of age story, beginning with the first integration problem: The data warehouse.

This is where ETL (extract, transform and load) technology cut its teeth, and as Devlin explains, it was very simple, practical, and, yes, technical, work.

“Although some enterprise modeling might have been done to figure out what the business meant when they said ‘customer,’ most of the real work was carried out at the database and even system levels to figure out if the six-digit customer number in one system could be matched to the eight-digit id in the next and what processing and conversions would be required to make it happen,” Devlin, the founder and principal of 9sight Consulting, writes.

And that pretty much sums up nearly two decades of integration work.

It wasn’t until the 2000s that enterprise information integration (EII) came along. If you’re not a technology person, don’t worry too much about the acronym — it’s now called data virtualization. Whatever you call it, as Devlin explains, it’s still a way of achieving “immediate integration,” as opposed to ETL, which is “prior integration” — meaning, you have to do the integration before the business can use the data.

Devlin actually sees these as “two disparate aspects of integration,” that “exist more on a continuum.” And, he adds, the problem is that this leaves us with data — not information.

Enter the third leg of Devlin’s trinity: concept integration. And like most things with the word “concept” in the title, it’s not quite so easy to explain as ETL or data virtualization. In fact, the piece becomes pretty non-specific after he introduces this next aspect of integration. I guess that’s why he’s writing a book about it.

He mentions that concept integration, or the integration of integration, requires modeling, knowledgeable professionals, and metadata.

“Concept integration involves knowledgeable professionals understanding and interpreting business meaning, how it has been represented in data stores and the processing that went into their population—on the fly and using text analytic tools, for example,” he writes.

Designing the tools to accomplish that, he says, is the domain of the vendors.

I guess that’s how it is with technology — it’s so much easier to explain and understand in retrospect than while it’s still emerging. Still, it’s important to understand those roots  — maybe more so when the technology is still evolving.

Devlin’s piece reminds us that integration’s Coming of Age has been primarily a technological endeavor. Maybe concept integration is the point at which we start to add human expertise back into the story.