IT metrics are like art. No one can seem to agree on what constitutes a good metric, but everyone seems to know one when they see it. And many people spend lots and lots of time studying them, discussing them and arguing about them, while a relatively small group of folks produce them.
There's a disconnect between IT and business when creating metrics, which I wrote about in November. IT tends to remain focused on data-intensive measures of existing technical capabilities, while business users would like to see more metrics that actually relate to how they do their jobs.
I cited some advice from Colin Fletcher, BMC Software's solutions marketing manager for BMC Atrium, who suggested IT departments should work with business users to create key performance indicators that focus on customer perceptions rather than hard data. Once IT knows what customers consider important, it can then establish baselines to illustrate the current status and begin working backward to identify the infrastructure issues associated with those KPIs. Focusing first on customer perceptions should result in a state where "you only quantify and measure what is really important," Fletcher said.
Another common issue: IT manages too many metrics or not enough. I addressed that topic last August and offered some great tips on striking the right metrics balance from Jim Quick of Diamond Management & Technology Consultants.
Fletcher's advice is echoed in a list of 12 characteristics of effective metrics written by TDWI's Wayne Eckerson. To achieve the first characteristic, "strategic," Eckerson says organizations "must start at the end point -- with the goals, objectives or outcomes you want to achieve -- and then work backwards," which sounds a lot like what Fletcher suggested.
Good metrics are also based around strategic objectives and designed to help organizations determine if they are on track to achieve their goals. And sorry IT, but "strategic" isn't a word most folks would apply to basic operational metrics. Those basic measures will be part of the broader metric, but they are a means and not an end.
I'm not going to reproduce Eckerson's entire list here. I encourage you to follow the link above and read it on his excellent Wayne's World blog. But here are a few of the other 11 characteristics I especially liked:
Also included in Eckerson's post are his thoughts on some of the reasons it's so tough to create metrics that accurately measure activities. First, inaccurate data yields poor results and creates employee mistrust. Organizations need to invest in data quality, an intimidating project that may be best achieved by breaking it into more manageable chunks and getting areas other than IT involved.
The way metrics are calculated also can produce misleading results. Eckerson offers the example of calculating productivity by dividing revenues by total number of workers. A rise in inflation can artificially boost revenues, meaning the metric will change even though workers weren't necessarily more productive.
And metrics may not accurately measure the intended objective. The example: Employee satisfaction may be measured using an absenteeism rate, but workers miss work for many reasons not at all related to satisfaction, such as health problems or issues with child care. Many organizations employ simple-to-measure metrics instead of ones that really yield insights about their performance, a problem I wrote about back in December.