IT’s Big Data Challenges Extend Beyond Size

2014 Big Data Outlook: Opportunities and Challenges

It’s hard to have a conversation in the enterprise these days without the topic veering toward Big Data. What is it? Where does it come from? And what are we supposed to do with it?

But despite the fact that none of these questions have clear answers yet, IT is still tasked with preparing to accommodate Big Data and then figuring out how to derive real value from it.

Part of the problem is the term “Big Data” itself. While large data volumes are a facet of Big Data, that’s not where the challenge lies. Rather, says IBM’s Doug Balog, it’s the need to accommodate the ‘variety, velocity and veracity’ that advanced analytics require that will give most managers fits. This will require not only bigger, more scalable infrastructure, but entirely new ways to collect, analyze and store data, which, from IBM’s perspective, will require advanced Power8 architectures married to powerful third-party platforms like Canonical and the various Linux distributions.

One way to make Big Data feel a lot less big is through automation, says StackIQ CTO Mason Katz. Already, large volumes are starting to overwhelm traditional systems-management platforms, so instead of fighting a never-ending battle for system optimization, try shifting the focus toward advanced apps that can manage reams of unstructured data on their own. At the same time, organizations should look for ways to incorporate Big Data into their underlying business models; otherwise, they will wind up with exabytes of analysis but no real way to leverage it.

Probably the biggest hurdle that IT will have to jump in order to capitalize on Big Data is the removal of legacy silo architectures, says MarkLogic VP Adrian Carr. One of the key facets of Big Data is the ability to gather information from disparate sources, but this ability is compromised if critical institutional knowledge is trapped amid dozens of incompatible platforms. Relational databases and data warehousing help in this regard, but the costs are high and they are generally not flexible enough to accommodate rapidly changing business environments. Many financial firms, he notes, are already turning toward advanced NoSQL platforms that provide the scale and flexibility to thrive in Big Data settings.

We also have to consider the likelihood that the biggest users of Big Data won’t be people, but applications. Start-ups like RelateIQ are drawing venture capital with plans of merging Big Data and SaaS to create a new breed of service called the Data-Driven Application. The idea is to give apps the ability to cull and analyze data on their own, and then apply that knowledge to primary functions like CRM and ERP. Ultimately, the company expects to transform the entire enterprise application stack in this way, effectively putting the fruits of Big Data analysis into the hands of average workers, rather than data scientists and top-level data and systems engineers.

We’re still in the initial phase of Big Data implementation, so it’s easy to let our imaginations run wild, but it’s safe to say that Big Data analytics will become a major force in the world economy over the next decade.

The challenge, though, will be to draw real value from the analyses, rather than just more data.