How Real Companies Analyze and Use Big Data

Share it on Twitter  
Share it on Facebook  
Share it on Linked in  
Slide Show

Big Data Analytics

The first steps toward achieving a lasting competitive edge with Big Data analytics.

A recent survey by TDWI offers insight into where companies are with Big Data, advanced analytics and the crossroads of the two: Big Data analytics. It also shows that despite all the hoopla over Hadoop, organizations primarily rely on that old standby - the enterprise data warehouse - to manage Big Data.


TDWI surveyed 325 respondents from their own ranks, as well as solicited responses from other sources, with 58 percent of the responses coming from corporate IT professionals, 22 percent coming from business sponsors or users and another 20 percent from those in consulting. Academics and vendors were eliminated from the survey results.


In other words, it's giving you a look at how Big Data is viewed, handled and used within real businesses.


There's a lot to digest in this 40-page report, written by TDWI's always-excellent research director, Philip Russom. The first thing that struck me is how many organizations are already doing Big Data analytics, which the survey equates to applying advanced analytics - "actually a collection of different tool types, including those based on predictive analytics, data mining, statistics, artificial intelligence, natural language processing, and so on" - to Big Data.


How many are actually already engaged in Big Data analytics? One-third of organizations are already experimenting with it.


The second thing that stood out - because I've heard this once before - is that analyzing Big Data requires a different approach than traditional analytics. You don't ask a specific question. Instead, you explore the data to see what it tells you:

Instead of "advanced analytics," a better term would be "discovery analytics," because that's what users are trying to accomplish. (Some people call it "exploratory analytics.") In other words, with big data analytics, the user is typically a business analyst who is trying to discover new business facts that no one in the enterprise knew before. To do that, the analyst needs large volumes of data with plenty of detail. This is often data that the enterprise has not yet tapped for analytics.

The report also points out that enterprise data warehouses are not only the main vehicle for managing Big Data and advanced analytics - it's also the preferred tool. Sixty-four percent say they're already using the enterprise data warehouse for Big Data analytics and 63 percent say it's their preference. You'd expect that, since enterprise data warehouses are the incumbent approach. Interestingly, whether an enterprise data warehouse is used for Big Data analytics is tied to how "pristine" organizations like to keep their data warehouse data - the more pristine, the less likely they were to handle Big Data there.


The survey also asked about analytics databases outside the enterprise data warehouse and found that most companies plan to reduce the number of data marts and operational data stores on one hand, but on the other, 28 percent of users are already adopting vendor databases designed specifically for Big Data analytics, and 30 percent say this is preferable.


Russom writes:

Users interviewed by TDWI described their use of new vendor-built analytic databases as being akin to an old-fashioned data mart, but with far greater data volume, detailed data, and data type diversity. Only time will tell whether new analytic databases will lead to the proliferation abuses we associate with data marts.

Interestingly, Hadoop is already in use by 24 percent of respondents, but only 30 percent expressed a preference for Hadoop and similar distributed file systems. Russom calls the adoption percentage a "respectable presence," but with reservations:

Given the open-source nature of Hadoop, TDWI suspects that many of these are simply downloads that are in experimental use. Hadoop is so heavily hyped at the moment that it is difficult to say whether its current experimental use will evolve into a permanent presence in IT.

Overall, though, the outlook for Big Data - and Big Data vendors - is oh-so-sunshiny. When asked about their view of Big Data, the overwhelming majority - 70 percent - said it's an opportunity for cutting costs and growing revenues. Only 30 percent labeled Big Data a problem.


This is a "best practice" report, so it's not just statistics.You'll also find a list of vendors that offer Big Data analytics tools and, on pages 34 and 35, a list of recommendations for pursuing Big Data analytics. All you need to do is register with TDWI for free to download the full document.


If you want to learn more about Big Data analytics, TDWI is also hosting a live webinar Thursday, Sept. 29, at 9 a.m. PT (that's noon for us East Coast residents). It'll feature Russom, as well as several speakers who will attend the upcoming TDWI World Conference and Big Data Analytics Forum, which will be held from Oct. 30 to Nov. 4 in Orlando, Fla. The webinar will cover how Big Data analytics compares to existing analytics and data warehouse practices, and other topics such as why the barriers to adoption. The event has a limited capacity, so you should register now if you're interested.