Confronting the Big Data Challenge

It looks like it’s going to be a big year for Big Data, although the prospect for even more massive data volumes isn’t exactly good news for many in the IT industry.

Gartner predicts that 2013 will see overall IT revenue topping $3.7 trillion, accounting for everything from software and services to smartphones and mainframe computers. Much of that spending will come as a rearguard action to counter the effects of Big Data, both in terms of building the infrastructure to accommodate it and devising the database and analytics tools to make sense of it.

For many observers, however, Big Data is like pornography: impossible to define, but you know it when you see it. For the typical enterprise, however, what exactly is the line between normal data volumes and Big Data? According to Dr. Maurits Kaptein of the University of Tilburg, any dataset that is too large to be analyzed in memory qualifies as Big Data, which is actually a pretty low threshold considering the amount of data that most organizations generate through on-line transactions and other Web-based applications. The key, then, is not to worry about building infrastructure capable of handling all this information, but honing your ability to weed out the useless bits in favor of data that has real value.

Indeed, if infrastructure is to have a significant role to play in the struggle to rein in Big Data, it will come from automation, not the actual analytics, according to GigaOm’s Derrick Harris. Just as robotics changed the shop floor by automating repetitive tasks, data automation will free IT techs from the day-to-day duties surrounding infrastructure and application management to concentrate on converting data into actionable intelligence. It’s all a matter of deploying the right infrastructure to take on the grunt work so human operators can pursue higher-order thinking.

And it goes without saying that what is Big Data for some is small potatoes for others. And that will likely be a problem for many small and medium-sized businesses that don’t have the means to acquire the Big Data systems currently hitting the channel but nonetheless are anxious to peer into their own volumes to see what they are worth. Perhaps if analytics developers had a more proportional view of Big Data, as SiSense’s Bruno Azria suggests, the industry as a whole would benefit both in terms of profitability and innovation.

In that regard, then, Big Data is both a challenge and an opportunity, not just for the enterprise but the development community that serves it. Consider it the data equivalent of the Old West: There is gold in them thar volumes, but it will take substantial willpower, and the right tools, to mine it.