Why the Internet of Things Data Will Spark an IT Revolution

In-Memory: Speeding Up Value by Using Operational Intelligence

In the beginning, experts said Big Data technologies would lead to the end of enterprise data integration, with data eventually moving into one, big in-memory or Hadoop system.

That was before the Internet of Things (IoT), with its never-ending stream of data. It seems the IoT is teaching Big Data humility.

A TechTarget article about a recent O’Reilly Media webcast includes this very telling quote from Mike Olson, co-founder and chief strategy officer for Hadoop distributor Cloudera: “It turns out machines are much better at generating data than you or I. It’s why big data is happening; it’s why industry is so quickly being transformed.”

He was joined in the webinar by Scott Jarr, who works for in-memory database provider VoltDB, in supporting a new motto for Big Data:

Loosely joined small systems that manage data where it’s born.

“Loosely joined” sounds suspiciously like the “loosely coupled” of SOA, but without all the baggage about protocols.

 That’s a significant shift for the industry, and the dual conversation between an in-memory expert and a Hadoop expert suggests that there’s a role for both in this Brave New World of data.

What happens in IT is often described as “revolutionary,” but often it’s actually evolutionary. The IoT may be the exception.  I don’t mean revolution as in Che Guevara hats and red sickles, of course. I mean revolution, as in “a sudden, extreme, or complete change in the way people live, work, etc.”

That’s because the IoT places two real demands on IT systems that, frankly, it is not prepared to handle. It requires managing Big Data in all its glory: high volume, high velocity, and high variety.

IT systems will need to adapt, and evolving — or simply adding on to the existing relational database management system architecture — isn’t going to cut it.

What does this mean, if not Hadoop or in-memory as the end-all, be-all? TechTarget writer Nicole Laskowski sums it up nicely: Architecture matters.

It turns out, even that simple statement will require something of a revolution, as Jason Bloomberg explains in a Forbes column.

Bloomberg is an EA expert who spent 12 years at ZapThink, a SOA consultancy firm. Frankly, the rise of service-oriented architecture was huge for enterprise architecture in general, promising to make EAs a major force in businesses.

Apparently, businesses and enterprise architects alike are still waiting for that eventuality. Bloomberg writes that while people have been successful in architecting enterprises, many are not actually EAs or even married to a particular EA framework.

Meanwhile, Bloomberg writes, actual enterprise architects spend their time documenting IT systems, with frameworks that “tend to become self-referential,” as Angelo Andreetto, senior EA for Zurich Insurance Group in Zurich, Switzerland, so kindly put it.

Could Big Data be what triggers business relevance for EAs? Bloomberg obviously thinks so, and sources other EA leaders to support his idea. For instance, he quotes Ken Griesi, a leading practitioner and thought leader in the EA field:

“With the move toward digitization and mobility, there has never been more data being generated about an enterprise than now. The introduction of Big Data Analytics is a game changer for EA. Truthful data provide EA with the pulse of the enterprise and its environment.”

That doesn’t mean EA will be revolution-free. In fact, Bloomberg’s theme is that enterprise architecture must reinvent itself, and that will most likely include removing “Miltons” (as in, the stapler guy from Office Space) from the ranks.

If you’d like to avoid Milton’s fate, or if you’d just like to learn more on how enterprise architecture needs to change, check out this TDWI Checklist Report, “Adopting Next-Generation Data Technologies into your Enterprise-Class Environments.” It published last month and is written by David Loshin, a data management and BI expert, TDWI instructor and president of Knowledge Integrity.