Better DevOps Through Log Management

For enterprises looking to make the most of their new DevOps environments, the importance of log management cannot be overstated.

Without logs, there is no way to track performance, analyze security flaws, monitor resource consumption, or manage the myriad other aspects of highly fluid, constantly changing data environments. In a way, then, DevOps and log management enjoy somewhat of a symbiotic relationship: DevOps fuels the need to make better use of logs, which in turn helps to continuously improve and streamline the application pipeline.

One area that will benefit significantly from improved log management is security, particularly as apps and services start to permeate the cloud, says Shira Shamban, data initiative lead at Dome9 Security. With traditional firewalls no longer an effective barrier to intrusion, many organizations are turning to configuration management solutions, anti-virus software, IDS and other techniques, all of which generate a large number of logs that can nevertheless result in blind spots unless there is a robust and scalable way to consolidate and analyze them. The caveat here, though, is that this is not something that can be managed as an afterthought or on a part-time basis, but must become a core objective of the DevOps environment.

Indeed, says Sumo Logic’s Scott Fitzpatrick, proper log management is the only way to ensure the security of code as it is continuously being pushed into production environments. When establishing a proper DevSecOps ecosystem, developers should be constantly writing code that logs data regarding relevant security events, such as authorization failure, input validation issues and the like. In this way, log creation becomes the foundation for a secure app, rather than an add-on. Even as their code enters a common codebase for testing and then into full production, security techs will have a clear audit trail through which they can track down any issues.

One potential risk to effective log management is the rise of containers. Sylvia Kalache, co-founder of the Holberton School, noted on Loggly’s blog page that even one process per container can dramatically alter the volume of hosts and the volume log. Therefore, it is crucial that logs be moved away from containers as soon as possible while maintaining real-time, or at least high-speed, analysis in order to correct any issues before the container itself is decommissioned. This will require a new container-optimized architecture and logging infrastructure, as well as a new way of managing and shipping apps.

As time goes by, even automated management systems may start to crack under the volume of logs being generated. This is where artificial intelligence can come to the rescue. According to Idexcel Technologies, a branch of AI known as Cognitive Insights (CI) is showing remarkable promise at matching human domain knowledge with log data and other input to form relevant insights into a wide range of critical IT issues. CI has the ability to monitor and register each and every log and then subject it to a range of queries to drill down into highly refined functions related to operational efficiency, customer fulfillment, risk reduction, and a host of others.

Log management is probably the quintessential example of jobs that are important but that no one likes to do. In a DevOps world, its importance rises to critical levels in direct proportion to the difficulty it presents in highly dynamic workflows.

And this is precisely why the enterprise needs to put log management front and center now, rather than retrofit it into an already unwieldy and risk-laden application ecosystem later.

Arthur Cole writes about infrastructure for IT Business Edge. Cole has been covering the high-tech media and computing industries for more than 20 years, having served as editor of TV Technology, Video Technology News, Internet News and Multimedia Weekly. His contributions have appeared in Communications Today and Enterprise Networking Planet and as web content for numerous high-tech clients like TwinStrata and Carpathia. Follow Art on Twitter @acole602.