Hadoop as the Next Application Development Platform

The Real Life of a Data Scientist

The big thing in Big Data this year might not be the data, but the applications.

It’s oxymoronic, I know, but it’s not the first time we’ve heard this. As I shared last year, SAP already contends that Main Street will adopt Hadoop as an application platform, rather than a data management tool.

Now, it seems that Hadoop vendors also are moving to an application development strategy, according to a recent Forbes column by technology consultant and writer Edd Dumbill.

“Hadoop needs more than just a data crunching engine and a small army of willing Java programmers. It must become an enterprise platform that supports application development,” Dumbill writes. “By the end of the 2013, the major Hadoop vendors had all formulated a platform strategy: be it the Cloudera Enterprise Data Hub, or Hortonworks Data Platform.”

Dumbill talks about a “data lake dream,” which is a lot like the Big Rock Candy Mountain. It’s a data-centered architecture, where distributed computing comes trickling down the rock and they hung the jerk that invented data silos. And to prove it’s more than just a developer’s dream, he points out that Google and Facebook developers “live the dream fully.”

That data lake dream is Hadoop-based, which is worth noting because, as we all know by now, Big Data technology is more than just Hadoop.

We’ll see how that works out, but for now, what’s more interesting is the Hadoop maturity model he outlines that would lead to this data lake dream.

Step one is pretty much the status quo, without Hadoop. Step two introduces Hadoop, right between the apps and the data warehouse and analytics engines.

Step three connects a majority of your apps to Hadoop, which by now is running the analytics on its own. A few apps still feed into Ye Olde Data Warehouse.

Step four: Hadoop is the center of the known universe, with all apps built on top of it. There are a few exceptions for maybe legacy apps or other “special” apps, and these still feed straight into the data warehouse.

Considering the vendors themselves are just now starting to deliver the capacity for application development on Hadoop, I’m going to guess that this dream lake is only marginally closer for many organizations than that Big Rock Candy Mountain.