Big Data: Not Just for Big Business Anymore
You hear it all the time: There simply aren’t enough trained data scientists to support the demand for Big Data analytics.
But here’s an interesting fact from TDWI’s best practices report on “Managing Big Data”: The data scientists aren’t really managing it now.
Actually, there’s an incredible range of job titles that manage Big Data. Out of 297 responses from 166 respondents (they could choose multiple options), only 6 percent said data scientists manage Big Data in their organizations.
Application developer, business analyst and systems analyst or systems architect also received 6 percent of the responses.
The top choice right now for managing Big Data is the data architect, at 16 percent, followed by the data analyst at 10 percent. Database administrators garnered 7 percent of the tally, with the rest going to a motley crew of IT jobs, including domain expert (seriously?) and ETL developer.
You could argue that that’s because of a shortage of data scientists. However, it’s worth noting that this particular survey attracted “unusually large percentages from the two kinds of organizations that are most prone to Big Data,” such as midsize to large Internet firms and corporations with $10 billion or more in annual revenue.
In other words, the very type of organizations you would expect would be able to attract data scientists. (Whatever that means.)
The data warehouse group also houses most Big Data projects, TDWI found, followed closely by central IT. Business units or departments are the next highest environments for housing Big Data, with about half as many responses as central IT.
TDWI also asked respondents which data management disciplines or teams were needed to manage Big Data.
Interestingly enough, BI and data warehousing skills topped the list of “strongly involved” skills, followed by data integration. DBA skills ranked third in the survey, with enterprise data architecture and data quality as the next most-popular choices.
So it’s interesting to see that most companies aren’t even using data scientists right now.
Nonetheless, Deloitte projects that over the next five years, U.S. organizations will need 190,000 data scientists or other people skilled in Big Data projects, reports Ziff Davis in a new white paper, “Analyzing Big Data: The Path to Competitive Advantage.”
How are we to reconcile this future need with the current status?
Neither report states specifically, but I think I see two clues:
- Right now, many organizations are managing fairly large — although not very large — data sets, by scaling up existing solutions rather than investing in Big Data-specific tools. I wrote about this yesterday, but the volume is expected to increase substantially over the next three to five years.
- Organizations aren’t overly enthusiastic about the results they’re getting. TDWI found that 64 percent rated Big Data management “moderately successful” in terms of supporting business goals. Another 24 percent said it’s not very successful. Only 12 percent labeled their efforts “highly successful.”
“Advanced analytic capabilities are necessary to make the data actionable,” warns Ziff Davis.
Perhaps that’s part of what’s keeping these projects from being “highly successful” now. If so, we may see those numbers shift as the problem becomes more acute.
