The Business Impact of Big Data
Many business executives want more information than ever, even though they're already drowning in it.
It's pretty clear to everybody at this juncture that we're exponentially dealing with more data than ever. Theoretically, more data should lead to better decision making. But while that's a nice idea in theory, the challenge we face today is that the tools we have available to analyze that data are still relatively immature, and the cost of storing all that data is growing beyond what the typical IT budget can support.
A new survey of 543 C-level and IT executives, conducted by the Kelton Group on behalf of the IT services firm Avanade, finds that email and other traditional enterprise applications still generate the most data and that, not surprisingly, business executives are struggling to cope with all of it.
But Avanade CIO Chris Miller rightly points out that things may get a lot of worse before they get better. With more sensors than ever starting to stream data back to the enterprise, coupled with a general rise in the use of mobile computing devices, there is going to be more data to analyze than ever.
The first order of business for many IT organizations is going to be to figure out how to store it all. Increasingly, we're seeing interest in high performance database systems at the high end to help process all this data, and open source frameworks such as Hadoop for managing it. Hadoop provides a low-cost mechanism for managing large volumes of data, while higher-end database systems provide the raw horsepower needed to analyze sets of data that are of special relevance to the business.
Of course, deciding what data to store where becomes the challenge. More often than not, there is data stored in an SQL database that hasn't been accessed in months or years. That's an expensive place to store data when there is a much more cost-effective option such as Hadoop or an open source database such as MySQL available.
What all this really means is that data management skills and technologies are going to be critical going forward. It's not enough to collect data, the system has to automatically recognize where best to store that data based on its relative value to the business. Once that's determined, then IT organizations have a self-evident framework for applying analytics applications to data that has the highest amount of value to the business.