Developing DevOps Metrics & KPIs

DevOps is not merely a technology upgrade but a change to business cultures, processes and models. So it shouldn’t come as a surprise that few organizations have a handle as to exactly what makes a successful transition.

One of the biggest mistakes is thinking that traditional performance metrics can give you an idea of what is working, and what isn’t, in a DevOps environment. While it is true that things like resource utilization, network traffic patterns, storage consumption and the like can provide insight into the IT infrastructure that still plays a crucial role in DevOps, the fact is that true performance depends on a whole lot more.

Going forward, it appears the enterprise will need to expand its monitoring capabilities into areas that border on the esoteric. According to experts recently polled by jaxenter.com, these can range from common metrics like uptime, response time and user satisfaction to things like how much weekend or overtime your DevOps teams are logging or how much sleep they are getting overall. As TCDI’s Eric Vanderburg notes, however, the best metrics will be the ones the enterprise defines for itself based on the objectives of its unique business model.

Determining what to measure is only one part of the challenge, however. Equally important is how fast you measure it, says Electric Cloud’s Anders Wallgren. In a recent post on Information Week, Wallgren pointed out that traditional cycle and response times are inadequate for today’s business environment. Instead, the enterprise should focus on building real-time feedback loops tied directly into automated systems in support of the continuous integration/continuous delivery paradigm. When everything from infrastructure, development, testing and deployment are defined as code, a monitoring process that takes weeks, days or even hours to collect and analyze data is a sure way to put yourself behind the competition.

The trick, then, is to determine the metrics to verify the business value you hope to get from a given app or service, says CollabNet CEO Flint Brenton. One way to do this is through “value stream mapping,” which incorporates data on the health status of the value stream, orchestration and performance of the tool chain and overall traceability to assess the entire software development lifecycle. With this data in hand, executives can gauge the value stream of one process against key performance indicators (KPIs), such as mean-time between failures, defect rate and change lead times, to establish a consistent measure of success.

But how will this work in an agile environment when the very definition of success can change from day to day? According to development strategist John A. Jaramillo, a good way to navigate this difficulty is to focus on the three key areas that have been driving business metrics for decades: quality, performance and budget. With these three goals in mind, the enterprise can accurately measure critical aspects of the DevOps team, the organization as a whole and customer satisfaction. At the same time, it can help determine if projects are being completed on time and on budget and whether the project has access to adequate resources. Once these broad strokes are completed, the organization can then fine-tune its metrics to meet increasingly specified objectives.

In all likelihood, metrics will continue to evolve as the DevOps era unfolds, particularly as technologies like containers, machine learning and the IoT gain in stature. And as analytics engines become more sophisticated, we can count on data from multiple metrics being parsed and combined in novel new ways to discern not only performance-degrading issues but opportunities that otherwise may go unnoticed.

Crafting a new agile business model is hard enough, even with a proper understanding of people and processes. It is near impossible when those tasked with overseeing the transition are kept willfully blind.

Arthur Cole writes about infrastructure for IT Business Edge. Cole has been covering the high-tech media and computing industries for more than 20 years, having served as editor of TV Technology, Video Technology News, Internet News and Multimedia Weekly. His contributions have appeared in Communications Today and Enterprise Networking Planet and as web content for numerous high-tech clients like TwinStrata and Carpathia. Follow Art on Twitter @acole602.