The Business Impact of Big Data
Many business executives want more information than ever, even though they're already drowning in it.
Daniel W. Rasmus is exactly the kind of guy you'd expect to be excited about Big Data. The intelligent systems engineer is a "strategist who helps clients put their future in context" using "scenarios to analyze trends in society, technology, economics, the environment, and politics in order to discover implications used to develop and refine products, services and experiences," his bio explains. He's a former analyst at Giga and Forrester Research, plus he directed Microsoft's Business Insights division, where, he says, he helped envision how people would work in the future.
In other words - exactly the kind of profession that you'd think would want to use Big Data sets to analyze past behaviors as a basis for gaining insight into the future.
But he's also exactly the kind of guy who can understand just how complicated, how treacherously hard to get right, Big Data will be.
By and large, these issues have more to do with human shortcomings than actual problems with the data. And that's why it's so important for CIOs and business leaders to read this piece. Before you invest in Big Data, you need to know what you're up against when it comes to effectively putting that data to use.
All of his points should give IT leaders and data specialist - including modelers - pause, but a few are of particular significance to business leaders. For instance, there's the temptation to use Big Data to confirm your own biases. Here's how he sees that happening:
One group of modelers advocates for one approach, and another group, an alternative approach, both using sophisticated data and black boxes (as far as the uninitiated business person is concerned) to support their cases. The fact is that in cases like this, no one knows the answer definitively as the application may be contextual or it may be incomplete (e.g., a new approach may solve the issue that none of the current approaches solves completely). Who wins these debates today may be meaningless because the implications have no near-term consequences, but companies that accept one approach over the other may be betting their firm's future on wishful thinking and unwillingness to admit what they don't know.
This isn't to say you shouldn't use Big Data; it's just to say that you should be very careful what you put together, how you interpret it and how much you risk when acting on the data.
That's hard for mere humans to do, it turns out - really, really hard:
We may well be able to connect all sorts of data and run all kinds of analyzes, but in the end, we may not be equipped to apply the technology in a meaningful and safe way at scales that outstrip our ability to represent, understand, and validate the models and their data.
His piece includes a number of different real and planned applications for Big Data. Ultimately, he's cautioning us to rein in the enthusiasm and approach Big Data cautiously, even gingerly. Instead of jumping to the big game-changing uses, he suggests a good first step is to focus on the identification, consolidation and governance of data.
That's always smart advice. But you don't need to run a query against a Hadoop data store to realize that, if history is any indication, that won't be what happens.