The Art of Creating Effective Metrics

Ann All

IT metrics are like art. No one can seem to agree on what constitutes a good metric, but everyone seems to know one when they see it. And many people spend lots and lots of time studying them, discussing them and arguing about them, while a relatively small group of folks produce them.


There's a disconnect between IT and business when creating metrics, which I wrote about in November. IT tends to remain focused on data-intensive measures of existing technical capabilities, while business users would like to see more metrics that actually relate to how they do their jobs.


I cited some advice from Colin Fletcher, BMC Software's solutions marketing manager for BMC Atrium, who suggested IT departments should work with business users to create key performance indicators that focus on customer perceptions rather than hard data. Once IT knows what customers consider important, it can then establish baselines to illustrate the current status and begin working backward to identify the infrastructure issues associated with those KPIs. Focusing first on customer perceptions should result in a state where "you only quantify and measure what is really important," Fletcher said.


Another common issue: IT manages too many metrics or not enough. I addressed that topic last August and offered some great tips on striking the right metrics balance from Jim Quick of Diamond Management & Technology Consultants.


Fletcher's advice is echoed in a list of 12 characteristics of effective metrics written by TDWI's Wayne Eckerson. To achieve the first characteristic, "strategic," Eckerson says organizations "must start at the end point -- with the goals, objectives or outcomes you want to achieve -- and then work backwards," which sounds a lot like what Fletcher suggested.


Good metrics are also based around strategic objectives and designed to help organizations determine if they are on track to achieve their goals. And sorry IT, but "strategic" isn't a word most folks would apply to basic operational metrics. Those basic measures will be part of the broader metric, but they are a means and not an end.


I'm not going to reproduce Eckerson's entire list here. I encourage you to follow the link above and read it on his excellent Wayne's World blog. But here are a few of the other 11 characteristics I especially liked:

  • Simple: Performance metrics won't work if folks don't understand them. Eckerson mentions a customer management unit that hosts forums in which it explains to field technicians how the company's repeat call metric works and how it might impact them. Not only that, but it has some of its strongest technicians offer tips on how they positively influence the metric.
  • Actionable: This one ties in nicely with the idea above, that employees must understand the corrective actions they can take in the event a metric trends downward. Of course, this requires organizations to empower their employees, a concept that many simply are not comfortable with.
  • Game-proof: Sadly, employees will sometimes try to circumvent metrics to make their lives easier. Eckerson offers the example of a organization that hires a market research specialist to audit its customer surveys so employees aren't tempted to tweak the results.
  • Standardized: Organizations must agree on common definitions of terms such as "sales" or "customers" if they want metrics to be understandable to different groups throughout the organization.


Also included in Eckerson's post are his thoughts on some of the reasons it's so tough to create metrics that accurately measure activities. First, inaccurate data yields poor results and creates employee mistrust. Organizations need to invest in data quality, an intimidating project that may be best achieved by breaking it into more manageable chunks and getting areas other than IT involved.


The way metrics are calculated also can produce misleading results. Eckerson offers the example of calculating productivity by dividing revenues by total number of workers. A rise in inflation can artificially boost revenues, meaning the metric will change even though workers weren't necessarily more productive.


And metrics may not accurately measure the intended objective. The example: Employee satisfaction may be measured using an absenteeism rate, but workers miss work for many reasons not at all related to satisfaction, such as health problems or issues with child care. Many organizations employ simple-to-measure metrics instead of ones that really yield insights about their performance, a problem I wrote about back in December.

Add Comment      Leave a comment on this blog post
May 10, 2010 3:00 AM lukas altar lukas altar  says:

Creating a good metrics needs knowledge and genius as well. it is an art that can be learned through practice and harwork.

Acai Max Cleanse

Nov 28, 2011 6:48 AM Julie Julie  says:

This is art indeed!

Nov 28, 2011 6:54 AM New York Architect New York Architect  says:

I agree! This is art, and I love art! I use to paint and sell my paintings. At first I was doing it for fun, after a while some friends asked me to paint different things for them and in the end people offered me money for my paintings. It has become more than just passion, painting and listening to old music are my true hobbies.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.