Beware the Data Collectors for They Are Us: Privacy and Big Data
In a move that sets the stage for the next evolution of Hadoop, Hortonworks this week announced it will acquire Onyara, a major contributor to an open source Apache NiFi project dedicated to creating tools for managing complex data flows across distributed Big Data applications.
Hortonworks will deliver its implementation of NiFi data flow software as Hortonworks DataFlow. The goal, says Tim Hall, vice president of product management for Hortonworks, is to enable IT organizations to build distributed Big Data applications that don’t require every piece of data to be moved into a centralized instance of Hadoop.
Instead, IT organizations can use NiFi code to collect data at a remote node and curate it at what Hall describes as the “jagged edge.” Once the data is captured by NiFi, Hall says Hadoop can remotely interact with that data to determine if it’s relevant enough to move it into Hadoop itself. The end result should be much less network bandwidth consumption by distributed Big Data applications in Internet of Things (IoT) environments, says Hall.
In effect, Hall says NiFi allows developers to more easily create distributed Big Data applications using a set of visual tools that are optimized to manage multiple flows of data in real time. Prior to the existence of NiFi, developers were forced to build these applications using tools that were designed for more sequential processes. As a result, a lot of complex programming was required to build applications involving multiple data flows, which made those applications both expensive to build and fairly rigid in terms of making any changes after the application is deployed. In contrast, Hall says NiFi allows IT organizations to more easily collect, conduct and curate large amounts of distributed data.
It may take a while before new classes of business process applications built on Hadoop and NiFi make it into production environments in the enterprise. But, it is certain that by this time next year, many of these applications that were previously too complex to build will be a whole lot more commonplace than they are today.
