Metrics

By Guy Higgins

I came across an interesting reference recently – Goodhart’s Law. Charles Goodhart observed, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” I think that’s a pretty good statement, but the phrasing that I saw elsewhere that started me thinking about it was, “When a measure becomes a target, it ceases to be a good measure.” Both phrasings mean the same thing – data can only tell you something important when the data are not managed or controlled.

Let’s look at an example (in the following true story, the names have been changed to protect me). I was working in a division of an organization that was struggling with its budget. The actual budget numbers (overhead, research & development, etc.) should have been considered in light of the work that needed to be done, financial health of the organization and the future organizational viability. The problem with which folks were struggling was an unacceptably large overhead burden that drove up rates and adversely impacted profitability. The overhead budget (which should have reflected the division’s needs) was (or should have been) a metric – data with which to gauge the division’s and organization’s current health and future prospects. The goal/aim/objective should have been to be profitable in the present and be well positioned for the future. But that darn overhead budget number was too big, so someone (not I) suggested that a portion of the overhead budget be transferred to the organizational headquarters’ budget thus shrinking the division’s overhead numbers. Okay, now the overhead budget for the division is fine, but it’s no longer a metric – it’s a goal, and the goal (formerly the metric) is being managed, and is no longer a valuable metric to help understand current profitability and future positioning.

What does this mean with respect to choosing metrics? Good question. I think that it means two things:

  • Leaders need to choose metrics very carefully. The metrics that they choose need to be (I will cavalierly use the term) independent. By independent, I mean that the metrics (the measure of some performance or quantity) need to reflect management action, but not be easily subject to being artificially controlled or managed directly. For example, inventory on hand may be an interesting number for a company that has embarked on a program of implementing lean practices, but it is a trivial effort to simply cut inventory on hand by demanding Just-In-Time (JIT) delivery from vendors. While JIT makes me look good, it simply transfers the problem to my vendors and may well increase my vendors’ costs and therefore my costs – not a good result. Idle inventory on hand is (in the vernacular of lean) waste, and what I want to do is reduce waste and improve productivity. That may include reducing inventory on hand, but only if that reduction improves productivity, so I may want to look for measures that reflect productivity and are less subject to non-value-added management gaming (managing the metric – not the goal). I was the US Navy rep in a joint program once in which the contractor was suffering crushing cost and schedule issues (I was finally able to make a reasonably large contribution to getting the program killed [he said patting himself firmly on the back]). The contractor’s solution was to adjust their program plan (they did this every three months or so for about a year). That certainly made their metrics look good but the program was still a basket case. Leaders need to manage to their strategic goals, not their plans or metrics.
  • Leaders need to resist the temptation to manage those metrics. This means that they understand what actions actually result in improvements in achieving their goals and are therefore reflected in the metrics. This requires upfront thinking, and it needs to be understood by everyone in the company/organization. I was the program manager once on a reasonably complicated program that included a great deal of software development. We tracked our software development progress down to the individual coder. It took significant effort to assure the coders that, if they fell behind on schedule, the metric would tell us (program leadership) that there was a problem and that the problem was most likely an issue of inadequate resources – somewhere. It also meant that we had to walk that walk and not merely apply that famous “find someone who can” solve the problem.

Choosing good metrics and then letting the metrics tell you how you’re doing is extremely important, and it’s hard – it’ll require some serious leadership. Leaders manage the company/program/project/effort – not the plan and certainly not the metrics.

Thoughts?

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *