Want a Big ROI for Big Data? Use Hadoop to Drive Real-Time Automation

Loraine Lawson
Slide Show

Looking Ahead at the 2015 Business Intelligence Landscape

Organizations that reap high return rates on Big Data projects do so by changing operational systems for everybody, rather than “enlightening a few with pretty historical graphs,” according to IT research and consulting firm Wikibon.

How do you do that? You stop using Big Data to drive “using the rear-view mirror.” Instead, you couple Big Data insights with in-memory technologies so you’re “driving with real-time input through the front windshield,” writes David Floyer, a former IDC analyst and co-founder and CTO of Wikibon.org.

Floyer’s lengthy piece on Big Data ROI goes into the technical details on how you piece this together. His technologist background really shows, though, so here are a few terms you’ll need to know to follow it:

Inline analytics. He capitalizes this, which threw me off, but it’s not a brand name. Inline analytics simply means you’re embedding the analytics into a process, usually to automate decision-making. For instance, supply chains use inline analytics to trigger alerts when there are disruptions. The key here is that inline analytics are in real time, or near-real time. Supply Chain Nation has a good read on that, if you’d like to know more. This is where in-memory data technologies (such as SAP’s HANA) and flash come into play.  

Deep data analytics. This is another term for Big Data analytics, but what’s different here is how the results are used. When we talk about using Hadoop for analytics, it’s often as an end in itself. Wikibon’s research found that successful organizations “operationalized the Big Data processes and findings into operational systems.” Translation: They used Hadoop or data warehouses to develop and support the algorithms for inline analytics.

Leveraging Big Data to support real-time operations “radically changed the way Big Data is monetized and managed,” Floyer writes.

That doesn’t mean you’re off the hook for data scientists, but it may mean you can pay for them. Floyer writes that a small team of operational experts or data scientists can use Big Data to improve the algorithms and data sources.


The article also includes a graph showing how the integration of streaming data works, an analysis of four in-memory databases, and a short case study showing how a bottling company used this approach.

If all of that sounds like too much for your organization, then you might want to consider deploying Hadoop for ETL.

“Often Hadoop is used as a cheaper and faster way of enabling the ETL process,” Floyer writes. “When Hadoop is used this way, the elapsed times for ETL can be reduced. The costs of the ETL process can be reduced significantly by utilizing open-source tools and databases.”

Loraine Lawson is a veteran technology reporter and blogger. She currently writes the Integration blog for IT Business Edge, which covers all aspects of integration technology, including data governance and best practices. She has also covered IT/Business Alignment and IT Security for IT Business Edge. Before becoming a freelance writer, Lawson worked at TechRepublic as a site editor and writer, covering mobile, IT management, IT security and other technology trends. Previously, she was a webmaster at the Kentucky Transportation Cabinet and a newspaper journalist. Follow Lawson at Google+ and on Twitter.

Add Comment      Leave a comment on this blog post
Mar 18, 2015 2:24 AM Sam Sam  says:
Great article! Thanks to share! Big data is one of the most used buzzwords at the moment. A big data architecture contains stream processing for real-time analytics and Hadoop for storing all kinds of data and long-running computations. A third part is the data warehouse (DWH), which stores just structured data for reporting and dashboards. A DWH is a great tool to store and analyze structured data. You can store terabytes of data and get answers to your queries about historical data within seconds. A real world use case of this is at BlueCrest (one of Europe’s leading hedge funds), which combines HP Vertica as DWH and TIBCO StreamBase to solve exactly this business problem. BlueCrest uses StreamBase as a real-time pre-processor of market data from disparate sources into a normalized, cleansed, and value-added historical tick store. Then complex event processing and the DWH are used as data sources to their actual trading systems using StreamBase’s connectors. More at https://intellipaat.com/hadoop-online-training/ Reply

Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.