Why IT Metrics Are Like Piecrust

Ann All

Creating effective IT metrics is kind of like making piecrust. It doesn't seem like it would be too hard, in fact it sounds downright simple. Yet many people go through lots of failed versions before getting it right, and some folks never manage to produce a good crust.


And so it goes with metrics. Chris Lockhart discusses a common problem in creating metrics, coming up with ones that aren't connected to any real business value, on his Chris on EA blog. His example: A former client devoted a good amount of time and effort to collecting and presenting monthly data on projects created using an internal system development lifecycle (SDLC), with the project management office cheering every time the numbers went up. Yet there were no accompanying numbers showing whether this reduced costs or resulted in other benefits.


Soon, Lockhart writes, the company realized it needed to ask how much variance was reduced by using the SDLC, and how much that saved the company. Did quality increases promote profitability? Was it about money saved, costs avoided or revenue increased? He says:

Once these questions were asked and measurements taken to answer them, it became apparent that the overhead of running their custom developed SDLC (including training and enforcement and checkpoints and governance) was greater than the measured dollar benefit of the program.

Ouch. There's yet another lesson in the ineffectiveness of creating metrics just for the sake of doing so and becoming more caught up in creating the metrics than in actually assessing what they mean.


This is my chance to trot out some of the great tips on metrics I've shared in the past.


In May I shared a list of 12 characteristics of effective metrics written by TDWI's Wayne Eckerson. Among my favorite characteristics from his list: simple, actionable, standardized, game-proof and strategic. In order to make their metrics strategic, Eckerson advised, organizations "must start at the end point-with the goals, objectives or outcomes you want to achieve-and then work backwards." Doing this would help avoid scenarios like the one offered by Lockhart in his post.


In another post, I mentioned the common problem of IT trying to find the right balance in its metrics and offered some great advice from Jim Quick of Diamond Management & Technology Consultants on how to do so. For example, one tip for IT organizations that manage too many metrics: Go down the list of reports and ask, "What do we do when we look at this report?" Eliminate those for which the answer is "nothing." And here's a tip for those with not enough metrics: Identify an inefficient behavior you want to change and develop metrics so you can make a fact-based case that will persuade people to change it.


I've also cited some suggestions from Colin Fletcher, BMC Software's solutions marketing manager for BMC Atrium, on how IT can produce metrics that are relevant to business user. Hint: First focus on customer perceptions rather than hard data.

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


Add Comment      Leave a comment on this blog post
Nov 3, 2010 8:34 AM Chris Lockhart Chris Lockhart  says:

First, much gratitude for the linkage and quotage (Not a word, but hey).

Second, great point regarding characteristics of metrics. If they were all simple, actionable, standardized, game-proof and strategic (among others) then I'd think we'd be much more able to discern what is worthwhile and separate the wheat from the chaff. In retrospect, I wish I had said just that instead of the hundreds of words I used instead.

Nov 3, 2010 1:42 PM mataj mataj  says:

Measuring intellectual work makes no sense.

Intellectual work is about solving problems, and the only known mathematical way of measuring that is Kolgomorov Complexity, which is not a computable function. Any metrics anyone can possibly conjure will therefore be inadequate. In order to be fair, the metrics would have to contain the entire knowledge of the field being measured, it would have to be on intellectually higher lever. This is quite OK for measuring the coal-shuffling, but not much else.

If people take inadequate metrics seriously (which they do if their salary depends on it), the result will always be a fiasco. The best example is probably programming.

The very 1st metrics in use, back there in the 1980s, was number of lines. The result: A lot of empty lines & comments.

Then, there were noncomment lines and/or size of compiled program. The result: A lot of lines that did nothing, and functions never called.

Then, there was Function Point analysis. The result: A lot of unnecessary functionality. Data were formatted, unformated, reformatted, unreformatted, written to, and read from disk a couple of times without any other reason but function point gathering. Programs became slow, bloated, and buggy.

Then, they said: OK, enough of that bloatware, let's concentrate on quality. Punishing bugs was out of the question for several reasons. Nobody would write programs anymore for fear of making a bug, which couldn't be found anyway, because no programmer would ever admit a mistake. And so, testers were rewarded for every bug they found. The result: Programmers and testers made a deal. Programmers produced bugs for testers to find, and reward was divided in half.

If you think this is bad, well... the consequences of metrics in healthcare can be much worse. When Margaret Thatcher was reforming the British public health system, she hired a mathematician from Rand Corporation to devise some point based metric. Before that, he was optimizing the number of megatons to be dropped on Russian cities to maximize the number of deaths. A Cold War stuff. Anyway, his system punished appointment waiting times. The longer the queue, the larger the number of negative points. The result: The doctors cured quick and easy cases first, and let the serious ones wait. Queues got shorter, of course, and there was also an additional bonus: Lots of seriously ill patients died before treatment, which decreased the workload&costs and improved treatment effectiveness metrics.

No matter how you turn it- the ultimate metric is the success of the organization as a whole. In order to achieve that, the managers must actually do some managing, not just calculate metrics. In order to manage effectively, they must -- oh, horror of all horrors -- have at least some clue about what they are managing.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.

Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.