Is Big Data an Island?

Big Data Is Creating Big Jobs: 4.4 Million By 2015

I suppose it’s smart that enterprises would want to test Hadoop by setting it up as a sandbox silo before they think about integrating it.

Still, I couldn’t help but be a bit disappointed that there’s so much misunderstanding of its potential, that one technologist spent a whole year just trying to get permission to bring Hadoop into the company, and only achieved it when he promised it would be “insulated and isolated.”

The unfunny punchline? He worked in a CTO’s office.

Gil Press, a contributor to Forbes, shared this and several other vignettes from last week’s O’Reilly Media’s Strata NYC, which merged with Hadoop World this year, making it, according to Press, “THE big data event of the year.”

I gather this isn’t an unusual problem, because Press goes on to say most of the vendors at the event spent a lot of time talking about why Hadoop is fit enough to be a peninsula, instead of an island.

But it seems even some vendors in the data space aren’t sure it’s ready for enterprise prime time. James Dixon, CTO of the BI vendor Pentaho (which does offer support for Hadoop), told Press that Hadoop’s performance needs to improve 10 times over for it to be competitive in the enterprise space.

As luck would have it, he thinks the Apache Hadoop NextGen MapReduce — aka YARN — will do just that.

Given the very discussion, I have to wonder how well vendors are doing with Hadoop sales at this point. After all, many, many of the solutions they’re offering are about adding connectors or integration points so existing systems can use the data you’ve processed in Hadoop stores.