Hadoop’s Path to Good Intentions

Seven Ways to Make Big Data an Actionable Opportunity

Most enterprises today are using Hadoop for “data preparation,” according to a Sand Hill Group survey of 135 IT and business professionals.

What that means is that Hadoop is primarily a “poor person’s ETL,” according to Matt Asay, vice president of business development and corporate strategy at MongoDB.

In a ReadWrite column, Asay argues that that’s about to change, pointing to the Sand Hill Group’s finding that shows advanced analytics will shift from a mere 8.9 percent of the use cases to 24.4 percent.

It’s worth noting that 17 percent already say they’re using Hadoop for basic analytics, and nearly 18 percent say the open source solution is already deployed for business intelligence.

The slide presentation of the Sand Hill Group survey is online, so I checked it out. It turns out, 44 percent of those surveyed say they’re still in the “exploration and education” phase.

That’s a sizeable chunk, and worth considering in light of this finding:

• Almost 36 percent of companies indicated the results were somewhat or significantly less than expected.
• Slightly more than 11 percent said Hadoop’s results were either somewhat or significantly better than expected.

What to make of all this? I offer two additional thoughts:

First, everyone may not agree with Gartner’s hype cycle approach, but it strikes me that these are pretty typical results for a technology that’s hit the trough of disillusionment. As Information Week pointed out, it will likely be two to five years before we really see how Hadoop performs.

Second: That said, it’s going to be tricky to access Hadoop as a single technology anyway. Gartner’s Merv Adrian contended in an October post that Hadoop is a platform, which means a long list of solutions is being built for and on Hadoop, including:

• Application performance management
• BI tools
• Databases, including in-memory analytics-focused databases
• In-memory data grid engines
• Search
• Stream processing
• Workload automation

And that’s just a sampling of Adrian’s list, which he acknowledges is “by no means exhaustive.”

So, while companies say they will shift Hadoop from 17.8 percent BI use to 14.1 percent, and advanced analytics from 8.9 percent to 24.4 percent, if Hadoop is embraced as a platform, we may see all uses rise — or sink, depending on how enterprises like it.