The IT environment is rapidly changing: New technology stacks emerge every year that serve billions of people worldwide and naturally have been targeted by malware writers. How can you quickly and effectively distinguish a network intrusion attempt from an expected and authorized event? As it turns out, Apache Hadoop is one emerging technology that improves your chances of detecting and stopping security attacks.

Organizations realize that just putting up walls around data is no longer enough protection. CIOs want to avoid security-related incidents having an impact on service-level agreements (SLAs), so they want to reduce the likelihood of a successful attack while being able to respond faster when one does occur. What is needed to do this is a deeper insight into the data being generated in order to identify threats – and that happens by monitoring and analyzing all events across the network in real time. This approach, however, results in the generation of large amounts of security-related data that must be stored and analyzed. In addition, increased regulations require storing and archiving security event data for longer time periods to comply with more stringent regulations. That's where the advantages of using Big Data technologies on a real-time Hadoop platform come in.

According to Ted Dunning, chief application architect at MapR Technologies, it's possible to get in front of attacks by analyzing all network event data with tools such as Apache Spark running on a real-time Hadoop platform, and to do so economically. IT professionals can build models that identify "normal" behavior thanks to the large scale of data made available to them. An understanding of normal patterns enables the models then to identify anomalous behavior. The anomalies signal potential security threats, and the combination of the Hadoop platform with Spark gives the high performance and scalability needed for accurate models as well as the speed needed to alert organizations to take action quickly, thus reducing risk.