The IT environment is rapidly changing: New technology stacks emerge every year that serve billions of people worldwide and naturally have been targeted by malware writers. How can you quickly and effectively distinguish a network intrusion attempt from an expected and authorized event? As it turns out, Apache Hadoop is one emerging technology that improves your chances of detecting and stopping security attacks.
Organizations realize that just putting up walls around data is no longer enough protection. CIOs want to avoid security-related incidents having an impact on service-level agreements (SLAs), so they want to reduce the likelihood of a successful attack while being able to respond faster when one does occur. What is needed to do this is a deeper insight into the data being generated in order to identify threats – and that happens by monitoring and analyzing all events across the network in real time. This approach, however, results in the generation of large amounts of security-related data that must be stored and analyzed. In addition, increased regulations require storing and archiving security event data for longer time periods to comply with more stringent regulations. That’s where the advantages of using Big Data technologies on a real-time Hadoop platform come in.
According to Ted Dunning, chief application architect at MapR Technologies, it’s possible to get in front of attacks by analyzing all network event data with tools such as Apache Spark running on a real-time Hadoop platform, and to do so economically. IT professionals can build models that identify “normal” behavior thanks to the large scale of data made available to them. An understanding of normal patterns enables the models then to identify anomalous behavior. The anomalies signal potential security threats, and the combination of the Hadoop platform with Spark gives the high performance and scalability needed for accurate models as well as the speed needed to alert organizations to take action quickly, thus reducing risk.
The Secret Weapon Against Malware
Click through for three ways IT pros can use Hadoop technology to get in front of security attacks, as identified by Ted Dunning, chief application architect at MapR Technologies.
SIEM
With a combination of a real-time Hadoop platform and advanced analytics, organizations can predict, identify and deter security threats in several different ways, including:
Security Information and Event Management (SIEM). Hadoop can be used to analyze large amounts of real-time data from network and security devices. For instance, a large U.S. regional bank that was running out of storage capacity on its SIEM infrastructure chose to replace its SIEM with Hadoop. The bank now has the dual benefits of ensuring adherence to SEC/FINRA regulations for newer data sources along with the deeper analytical capabilities that machine learning on Hadoop provides using those data sets.
Network Intrusion Detection
Network traffic can be analyzed in order to detect and report suspicious activity or intruders. As an example, a globally managed security services provider harnesses Hadoop’s cost-effective distributed storage and breadth of open source ecosystem such as stream processing, SQL processing and machine learning to deliver new real-time threat detection services to its customers.
Fraud Detection
Hadoop can be used to perform anomaly detection on larger volumes and varieties of data to detect and prevent fraudulent activities. In one case, a large regional bank utilizes Hadoop to predict phishing behavior and payments fraud in real time to minimize impact on operations. The bank can run detailed analytics and forensic investigations in minutes compared to hours and update its predictive models in days instead of weeks.
