A colleague and I were talking a bit about Big Data recently, and she seemed a bit dismissive about Big Data. She expressed skepticism about actual, deployable technology for Big Data, and overall seemed to view it as more hype than reality in terms of actual deployments.
Then again, she's not in IT. But you probably already guessed that because CIOs are all too familiar already with the harsh reality of Big Data.
In fact, what I'm hearing over and over is that most CIOs of large organizations are well aware if they have a Big Data problem, and they're eager for solutions. I'm sure that will vary to some degree by the size of the organization, although certainly even small companies can have Big Data, depending on their industry.
Back in February, I wrote a piece for IT Business Edge's sister site, EnterpriseAppsToday, about eight business-changing ways companies are using Hadoop. Companies are using Hadoop to evaluate possible points of failure in the power grid, make smarter decisions about credit risks and even build their own internal clouds.
Most of the examples I found used Hadoop for analytics, but as Gigaom's Derrick Harris points out, there are broader, more innovative uses, including:
The federal government is betting Big Data will be the foundation for major innovation, too. It's investing $200 million in Big Data projects, such as:
Still, I can see why my colleague would be skeptical. After all, we all know there's a shortage of data scientists to help you analyze Big Data and IT staff to design and support it.
But that's only a small portion of the story. The truth is, vendors are offering new Big Data connectors and add-ons for analytical tools every day. Systems integrators, consultants and, of course, vendor services are beginning to offer help for Big Data projects. You may soon even be able to rent a Big Data model or subscribe to a Big Data SaaS.
So, while I understand the skepticism, what's being done with Big Data fascinates me, and the potential astounds me.
Sometimes, we're so amazed by technology, we forget what it can't do yet. For instance, I'd love to be able to scan in police reports - which are generally hand-written - and be able to analyze the reports for specific words. But right now, analyzing scanned images of handwriting - unstructured data at its most frustrating - isn't feasible, or at least not in any significant way.
I think it'd be fan-tabulous if scientists could analyze all the research ever produced on cancer and come up with answers.
While I'm frightened by the idea of an autonomous drone army and the ramifications of a war where one side has no casualties, it's thrilling to think Big Data could provide the AI foundation for Rosie the Robot.
Granted, some of the goals may be the stuff of dreams, but the technology is very real. It's still evolving in terms of its usability, but it is, nonetheless, being put to use in the real world, by real people, now.
And while some of the research may be the stuff of dreams, the technology is very real. It may be still evolving in terms of its usability, but it is nonetheless being put to use in the real world.