Researchers at the University of California-San Diego have come up with an interesting mental image to explain the amount of data processed by the world's computer servers in 2008. They put that number at 9.57 zettabytes of information.
Nine Predictions for the Analytics Industry in 2011
The competitive gap between analytical innovators and those who do not invest in analytics will widen over the coming 12 months.
But that's hard for our feeble human brains to process, so they put it this way: It's the equivalent of stacking Stephen King novels to Neptune and back 20 times. And they say that figure is rising 30 to 40 percent a year, reports The New York Times.
The San Diego Union quotes Roger Bohn, a UCSD technology management professor who co-authored the study, saying:
This is subterraean information. We see almost none of this. These are computers talking to each other at a level that is invisible even to people inside a corporation.
Forty-four percent of the data was in transactions such as issuing an invoice, paying a bill or checking a stock portfolio. The rest included Web services and office applications. The study estimated that enterprise server workloads are doubling about every two years.
The problem is that there are not enough human programmers to find more sophisticated ways to crunch data. The Times quotes Chaitan K. Baru, a senior scientist at the San Diego Supercomputer Center (SDSC), which is affiliated with the university, saying:
The growth rates are enormous, and they're only touching the surface of what's possible. You need people who can come up with those clever algorithms.
The university and the SDSC have announced plans for a new partnership to bring together industry and university research on better ways to manage Big Data.
According to Michael Norman, SDSC's director:
We are entering an era of data-intensive computing, where all of us-academia, industry, and government-will be faced with organizing, analyzing, and drawing meaningful conclusions from unprecedented amounts of data, and doing so in a cost- and performance-effective manner.
That sounds like a huge job opportunity to me, not only for managing the data, but also interpreting the data. People with skills in Hadoop and other Big Data techniques are in hot demand, as are analytics professionals. And though there are fewer data center jobs and more is being automated, someone still has to manage all that hardware.