Measure – Matter – Measure

By Guy Higgins

Last week, I came across an aviation article that urged airlines to adopt AoA (Angle of Attack)* as one of the “metrics” monitored by pilots to ensure proper airplane performance. AoA is very useful because it is self-compensating for airplane weight and remains an accurate indicator of wing stall. In fact, for practical purposes, the AoA for maximum range airspeed is always the same even though the actual airspeed varies with airplane weight. The same is true for maximum endurance airspeed and stall airspeed. AoA is a seriously cool, real-time, self-compensating metric for pilots.

Okay, fine – what has that got to do with non-aviation (which is a very large domain – sort of like non-dandelions)? To answer that, let’s look at the title of this post – Measure – Matter – Measure. We’ve all heard that what gets measured gets managed, and I think that is certainly true. The question I want to explore is whether or not what gets measured actually matters.

In addition to the article on AoA, I read a rant about metrics being imposed from the top down – the rant proclaimed, loudly, that such imposition of metrics was stupid. The authors asserted that the people doing the work should select their own metrics. If they do not choose metrics themselves, they will find themselves managing things (or at least reporting things) that they don’t control or which don’t matter to them. I agree, but…

The AoA article urged airlines (not pilots) to adopt the use of AoA. If AoA is such a useful indication, why aren’t the pilots already using it routinely? Why don’t the people doing the work (in this case the pilots) choose such a good metric? Lot’s of answers – all real, and none very good:

  • Because pilots don’t use AoA, airlines (and private pilots) seldom keep the AoA sensor calibrated, and it becomes inaccurate (which results in pilots not using it – the famous circular argument).
  • Really believing in AoA requires a greater understanding of aerodynamics than does something more intuitive – like airspeed – as an indication of flight condition.
  • AoA use isn’t emphasized as strongly as airspeed as an indication of flight condition during flight school.

So, what happens if the airlines do begin to emphasize AoA and force that metric on the pilots? From my experience (several times in my Navy career, I operated in situations where the use of AoA significantly improved our mission performance), once pilots experience using AoA routinely (and the system is kept in a calibrated condition), they love it – it’s simple and it’s effective.

In those cases, the “imposition” of metrics from the top worked very well. What about the imposition of metrics that are meaningful at the top level of an organization but not lower in the hierarchy? When that happens, that great answer, “It depends,” applies. When a leader is considering imposing metrics, she should ask herself the following questions, Is the metric something useful to the people who will be collecting and reporting it? Do they have the authority and resources to control the value of the metric? Unless the answer to those questions is yes, the metric is not only meaningless but is an additional burden and a waste of time.

During my Navy career, we kept track of all sorts of metrics. One of those metrics was airplane Full Mission Capable (FMC) rate (FMC means that the airplane is safe to fly and all the systems are working properly). FMC rate was viewed as critical by senior commanders, and, as the Maintenance Officer, I had a significant degree of control over that number. But “shift” happens and one of our sister squadrons experienced an accident that damaged a wing flap forcing the removal of the flap for depot-level repair. Obviously that airplane was “down” for the duration (between two and three months as I recall) and the squadron should not have been able to report an FMC rate any greater than 88 percent, but they “managed the metric” and regularly reported 100% FMC during that period by “creatively” interpreting the rules regarding airplane condition reporting. They managed what was measured (FMC rate) – not what mattered (how many missions they fulfilled). I’m certain that the squadron Commanding Officer, had he had the flexibility to choose his own metrics would have chosen mission success rate over FMC rate. Certainly, he would have chosen mission success rate over the rate at which his squadron gave blood (the monthly Vampire award was presented to the squadron with the highest blood donation rate – again, I kid you not).

This gets back to metrics – what should they be and who should choose them? I think that the right answer has to include:

  • The metric must capture something that actually matters.
  • The thing that matters must be under the control of the person responsible for the metric.
  • Metrics should not be imposed from above without ensuring that it actually contributes to understanding and managing performance at the level being directed to use the metric (AoA works for the pilots even though “imposed” by people flying LMDs (that would be large mahogany desks).
  • The metric should not be subject to management separate from the thing that matters (managing the FMC rate independent of the actual number of airplanes available).

Interestingly enough, AoA satisfies all of those conditions. It captures something that matters; the pilot can exercise control of the airplane to keep AoA at the right value; even though imposed from above (by the airlines in the article), it does contribute to understanding and improving performance; it is not subject to “gaming” – it can’t be changed or controlled without actually controlling the airplane.

So, what are the AoA-like metrics for you?

* For non-aviation buffs, AoA is the angle between the free airstream (the direction of airflow before being disturbed by the airplane) and the mean chord (the line between the leading and trailing edges) of the wing. The greater the AoA, the greater the lift generated up to the point where the angle is so great that the air no longer flows along the surface of the wing, but “separates” from the wing surface and the wing stalls (stops generating lift – this is a bad thing). You can see all kinds of neat images of AoA indicators here.


Leave a Reply

Your email address will not be published. Required fields are marked *