Want a Big ROI for Big Data? Use Hadoop to Drive Real-Time Automation

Looking Ahead at the 2015 Business Intelligence Landscape

Organizations that reap high return rates on Big Data projects do so by changing operational systems for everybody, rather than “enlightening a few with pretty historical graphs,” according to IT research and consulting firm Wikibon.

How do you do that? You stop using Big Data to drive “using the rear-view mirror.” Instead, you couple Big Data insights with in-memory technologies so you’re “driving with real-time input through the front windshield,” writes David Floyer, a former IDC analyst and co-founder and CTO of Wikibon.org.

Floyer’s lengthy piece on Big Data ROI goes into the technical details on how you piece this together. His technologist background really shows, though, so here are a few terms you’ll need to know to follow it:

Inline analytics. He capitalizes this, which threw me off, but it’s not a brand name. Inline analytics simply means you’re embedding the analytics into a process, usually to automate decision-making. For instance, supply chains use inline analytics to trigger alerts when there are disruptions. The key here is that inline analytics are in real time, or near-real time. Supply Chain Nation has a good read on that, if you’d like to know more. This is where in-memory data technologies (such as SAP’s HANA) and flash come into play.  

Deep data analytics. This is another term for Big Data analytics, but what’s different here is how the results are used. When we talk about using Hadoop for analytics, it’s often as an end in itself. Wikibon’s research found that successful organizations “operationalized the Big Data processes and findings into operational systems.” Translation: They used Hadoop or data warehouses to develop and support the algorithms for inline analytics.

Leveraging Big Data to support real-time operations “radically changed the way Big Data is monetized and managed,” Floyer writes.

That doesn’t mean you’re off the hook for data scientists, but it may mean you can pay for them. Floyer writes that a small team of operational experts or data scientists can use Big Data to improve the algorithms and data sources.

The article also includes a graph showing how the integration of streaming data works, an analysis of four in-memory databases, and a short case study showing how a bottling company used this approach.

If all of that sounds like too much for your organization, then you might want to consider deploying Hadoop for ETL.

“Often Hadoop is used as a cheaper and faster way of enabling the ETL process,” Floyer writes. “When Hadoop is used this way, the elapsed times for ETL can be reduced. The costs of the ETL process can be reduced significantly by utilizing open-source tools and databases.”

Loraine Lawson is a veteran technology reporter and blogger. She currently writes the Integration blog for IT Business Edge, which covers all aspects of integration technology, including data governance and best practices. She has also covered IT/Business Alignment and IT Security for IT Business Edge. Before becoming a freelance writer, Lawson worked at TechRepublic as a site editor and writer, covering mobile, IT management, IT security and other technology trends. Previously, she was a webmaster at the Kentucky Transportation Cabinet and a newspaper journalist. Follow Lawson at Google+ and on Twitter.