The Business Impact of Big Data
Many business executives want more information than ever, even though they're already drowning in it.
We're in the primordial soup days of Big Data. Every week, new solutions and partnerships crawl out of the muck to claim dominion in this new, exciting era.
The fact is even the best data scientists and engineers are still trying to figure out Big Data and how it fits in with the existing IT infrastructure. That's why there are so many startups swimming to Big Data, according to a recent CNET article.
"We're in the early days of the process of figuring which [database] engines match best to which workloads," Andy Palmer, co-founder of database company Vertica Systems and former CIO of Infinity Pharmaceuticals, told CNET at a recent Big Data event organized by the Massachusetts Technology Leadership Council.
You may wonder whether Big Data will eventually be subsumed into existing Big Vendor offerings. That happened a fat decade ago, GigaOm recently pointed out, when new database companies popped up around the idea that relational databases couldn't handle objects well. Eventually, the article notes, big database vendors "pushed and shoved at least some object capabilities into their databases, and those smaller companies disappeared."
But many business veterans in the database market say that won't be possible this time, including MIT Professor Michael Stonebaker and Andy Palmer, both long-time entrepreneurs of new database systems.
Certainly, the big vendors like Oracle and IBM don't seem to be headed in that direction - they, too, are joining the market with products specially designed for Big Data.
"What's different about big data is that it's driven by the Web and the Internet," Kelly Stirman, vice president of customer solutions at Hadapt, told CNET. "All the Web companies tried to use Oracle [databases] to solve their problems but eventually gave up."
Predictably enough, there are those who disagree.
Infostructure Associates analyst Wayne Kernochan, who has 20 years of experience in infrastructure software, recently wrote a cautionary tale about Big Data solutions and their role in the enterprise.
Kernochan argues that Big Data is being oversold and as a result, he contends few people understand its limitations. He predicts Big Data will become a supplement, not a critical part, of enterprise architectures.
" Handling Big Data is likely to require a careful mix of relational and non-relational, data-center and extra-enterprise business intelligence, with relational in-enterprise BI taking the lead role. And as the limits to parallel scalability of Hadoop and the like become more and more evident, the use of SQL-like interfaces and relational databases within Big Data use cases will become more frequent, not less," Kernochan said. "Therefore, I believe that Hadoop and its brand of Big Data will always remain a useful but not business-critical adjunct to an overall business intelligence and information management strategy."
My sense is that Big Data experts would disagree that relational databases would be usurped by Big Data. I do think they'd disagree with him about its business-critical role.
To be honest, Kernochan's piece delves into some very technical areas, and he's drawing on computational logic he worked out in college, which makes it difficult for the average reader, myself included, to follow. Techies should enjoy it, though, and will find his discussion on NoSQL's limitations and ties to relational databases a unique take on the topic.
That said, I will say he reaches some conclusions I can disagree with - specifically, his statement that Big Data is "effectively out there in the cloud' and therefore outside the usual walls of enterprise data centers." There are internal use cases for Big Data, since many organizations do have their own stores of Big Data, whether it's from sensor information or government agencies combining data.
There's also some early indication that he may be wrong about "mini-Hadoops," or Hadoop use cases, popping up within business divisions - but I digress.
For most of you, the bottom line is this: Big Data solutions are still emerging and evolving. At this point, it's hard to tell what organisms will survive, thrive or perish as they hit the hard shores of the enterprise market. Fortunately, many of your options are open source and most enterprise app vendors are now offering Hadoop plug-ins that will make it easier to use. Still, know that we're in the early days - possibly more early than we can even appreciate - so focus more on proven use cases and what your IT staff has the skills to handle than on marketing hype.