Data, Legacy Systems, and the New Age of Real-Time Architecture

Drivers and Challenges of Enterprise Integration Revealed

When I began covering integration and data, it really bothered me that there were so many integration problems. I mean, if I can get online and access information from a server in Hong Kong, why is enterprise IT such a mess?

I wonder how many business users feel the same way—frustrated by the silos and puzzled as to how servers around the world can serve up information via the web, but they can’t get a consolidated report due to application or political divisions within the company.

I eventually figured it out: The mess evolved from applications that at the time couldn’t connect to other applications. It was nobody’s fault and yet everyone was to blame.

If you’re a business leader or user frustrated by the data-sharing difficulties, then I recommend you check out Eric Kavanagh’s recent Inside Analysis piece, “Accidental Architectures and the Future of Intelligent Networks.”

The title sounds foreboding, but it’s actually a very readable and short history of information management and “accidental architectures.”

“Not everything happens for a reason in the world of information management,” begins Kavanagh. “More often than not, a company’s information architecture has grown and evolved organically, like a sort of digital mycelium, spreading underground for years, ultimately providing the infrastructure for all manner of analytical insights to blossom somewhere down the line.”

We often talk about evolution like it’s a plan where only the strong survive. That’s actually misleading: Evolution isn’t about the strongest or best overall; it’s about how generations of organisms adapted to their environment.

As we know too well, what’s best historically doesn’t always mean it’s the strongest in the current environment. For instance, our bodies retain calories as fat, a trait selected only because we couldn’t control our food supply. Now, for many of us, it’s a burden more than an advantage.

Likewise, as my evolutionary psych professor used to say, accidents happen even to the strongest and best. It is very possible that we’ve lost some wonderful traits over the years due to natural disasters and freak accidents. The biggest and best DNA won’t keep you from dying in a volcanic explosion.

IT systems aren’t that different, Kavanagh points out. Over the years, IT and the business adopted solutions that were available and that best suited the immediate need.

As technology changed, they adapted, adding on and amending as they could, but seldom ripping and replacing the entire system. The result is more Winchester House in design than a carefully planned grid.

He explains everything—why politics determines who gets data more than business priorities, the role of enterprise information integration and then data federation, a.k.a. data virtualization.

But Kavanagh is providing more than a history lesson here. His thesis is that we’re entering a new era for information management: the Age of Real-Time Architecture.

‘The speed of NoSQL engines (DataStax boasts a million writes per second), coupled with parallel processing and multi-core chips, has opened the door to what might just be called a new age in information architectures. This is no small thing.’

Even if you know the history of information management, this is a worthwhile read because he discusses some of the new players and new staples of the emerging IT architecture.

One aspect of this emerging new information management age is predictive analytics. If you’re interested in that specifically, you might also want to check out this Computerworld column, written by the vice president of decision management at BPM vendor Pegasystems. He discusses mostly vendor-neutral challenges, such as data quality, model complexity and data volume.