Big Data: Eight Facts and Eight Fictions
Only 8 to 10 percent of organizations have actually spent any money or time building Big Data applications or systems, according to a recent article in Datanami. But does that mean we’re all being conned about the growth of Big Data?
Probably not. Even though that 8 to 10 percent figure was consistent when Datanami looked at surveys by Gartner, The Data Warehouse Institute (TDWI) and data integration vendor Talend, that particular statistic offers only a small view of the Big Data picture.
As the article goes on to explain, there are other reasons to believe Big Data is still a major issue for organizations. In fact, the same Gartner study also found 64 percent of respondents either are investing or have plans to invest in Big Data technology this year. Other surveys show similar results.
Of course, jumping from 8 to 10 percent to over 50 percent adoption would be a huge shift, even over three years.
All of which made me wonder: What’s really going on with Big Data?
“The State of Big Data Management,” a TDWI report released last week, drills down a bit farther to shed light on these conflicting results.
It’s important to note upfront that TDWI recognizes “unusually large percentages from the two kinds of organizations that are most prone to Big Data,” midsize to large Internet firms and corporations with $10 billion or more in annual revenue. In fact, out of the 461 responses used, 189 were by respondents who have experience managing Big Data.
Still, the report provides a clearer picture of what’s going on with Big Data because it examines the issue from several angles.
It turns out, even though adoption of dedicated Big Data management solutions, such as Hadoop systems, may be low, organizations are still managing and using Big Data.
Just as with other surveys, the TDWI report did find that only 10 percent have deployed a dedicated Big Data management solution. (TDWI notes that figure is consistent with the 11 percent of respondents who say they already have a BDM solution in production, but I’m not sure what’s different about that additional percentage point.)
And like other surveys, the results show that more organizations plan to deploy Big Data solutions.
What’s surprising is their timeline. The TDWI report found the current 10 percent deployment number is expected to double within the next six months. And, just as with Gartner’s survey, about half hope to have deployed a BDM solution within three years.
But here’s another interesting part of the equation: Half of organizations report they’re already actively managing Big Data. In fact, most organizations report data in the 10 to 99 terabyte (TB) range.
Huh?
It turns out, a quarter of organizations have Big Data in the form of structured and relational data. That means they’ve been able to manage these Big Data sets by scaling up pre-existing applications and databases, TDWI states.
Now the 10 to 99TB range isn’t what you’d call “very large,” the report notes. Those levels are more in the 100TB and up range, but within three years, TDWI says that range will be the norm. Twenty-three percent say they’ll even break the one-petabyte barrier in that timeframe.
It will be intriguing to see if the existing tools will continue to scale well for those organizations. The report is available for downloading with free registration at TDWI’s site.
