Metric Dysfunction

The world is obsessed with quantifying every possible outcome, with the hope that creating a metric will provide objective, unbiased feedback on organizational performance.

The usefulness of metrics depends on how they are used. Metrics are not intrinsically good nor bad.

Context is everything.

If used wisely, they provide unambiguous clarity.

If used poorly, they replace judgment and expertise with false precision, misaligned incentives, and irrelevant quantification.

The intent behind metrics is legitimate. The implementation, however, fails to account for the drawbacks and incentive distortions inherent in trying to track complex activities.

First, we need to understand the limitations of metrics.

Second, always incorporate human judgment when assessing an outcome-based metric.

Metrics, taken at face value and without interpretation, do more harm than good.

Jerry Muller, author of The Tyranny of Metrics, describes the damage and insanity of metric obsession. Jerry shares many examples of how metrics have disrupted organizations.   

Let’s first look at all the ways metrics have gone astray.

Doctors

Measuring doctors based on patient outcomes makes sense on the surface but has a nasty side effect. Prioritizing outcomes encourages doctors to only take the easy cases and decline the hard ones.

This is called “creaming.”

Jerry Muller explains:

The superior surgeon uses his superior judgment to steer clear of any situation that might test his superior ability. That is, he avoids difficult cases as a way of maintaining his success rate. A classic strategy of “creaming,” that is, avoiding risky instances that might have a negative impact on one’s measured performance.

Numerous studies have shown that when surgeons, for example, are rated or remunerated according to their success rates, some respond by refusing to operate on patients with more complex or critical conditions.

Hospitals

Hospitals can be punished if wait times exceed a certain length. What’s the solution? Just keep the patients parked outside so the clock doesn’t start and then admit them when you can ensure responsive treatment.

From The Tyranny of Metrics:

In England, in an attempt to reduce wait times in emergency wards, the Department of Health adopted a policy that penalized hospitals with wait times longer than four hours. The program succeeded—at least on the surface. In fact, some hospitals responded by keeping incoming patients in queues of ambulances, beyond the doors of the hospital, until the staff was confident that the patient could be seen within the allotted four hours of being admitted.

Academic Institutions

Measuring student success by GPA? Just lower the bar. Everyone gets an A!

Trying to maintain a low acceptance rate to showcase exclusivity? Incentivize applications that have no chance of acceptance, thereby boosting the denominator and depressing the acceptance rate.

Airlines

Want to maintain a great on-time record? Just pad the flight schedules with an extra hour. Now flights are way ahead of time even though the actual flights may still be arriving later than otherwise would have.

Military Strategy

In Vietnam, body count was used as a metric to judge how well the U.S. was “winning.” And we all know how well Vietnam worked out for the U.S.

Muller elaborates:

As secretary of defense in charge of prosecuting the war in Vietnam, McNamara championed the metric of “body counts” as a purportedly reliable index of American progress in winning the war. Yet few of the generals in the field considered the body count a valid measure of success, and many knew the counts to be exaggerations or outright fabrications.

Metrics are favored most often by those who are the most ignorant: the bureaucrats and leaders who have no idea what’s happening on the ground, whether in war or in organizations.

Police

Want to show improving crime statistics? There’s the hard way – catch more bad guys. Or the easy way - just manipulate the focus and reporting.

From The Tyranny of Metrics:

Ed Burns, a former Baltimore police detective for the homicide and narcotics divisions—best known as co-creator of the HBO series The Wire—has described the process of “juking the stats,” by which police officials could orient the activity of the department toward seemingly impressive outcomes. As a detective in the narcotics division, Burns sought to meticulously build a case against top drug lords. But his superiors were uninterested in that prospect, which was consuming manpower and would take years to produce an arrest. They were interested in enhancing the metrics, and since arresting five teenagers a day selling drugs on street corners yielded better statistics than arresting a drug king-pin after a multiyear investigation, they favored the course that quickly produced the higher numbers.

Investing/Business

Want to measure employee productivity? Measure how many hours they worked, even though you have no idea what they are really doing with that time.

Rewarded for low loss rates, no defaults, or avoiding “problem” securities? Easy, just by treasuries or other low risk investments. Your loss rates will look fantastic, even though your total return will lag a more sensible risk-taking strategy.

Rewarding your team based on how many meetings they take, securities they evaluate, or managers they diligence? Great, you’ll get lots of quantity, with no idea on quality.

Judging team members’ attentiveness based on how quickly they respond on Slack or email? Great, they will respond fast, at the expense of constant distraction and zero ability to focus deeply for extended periods.

Telling your managers to get invested quickly? They certainly can, but it’s good performance we want, not just money invested.

LP’s obsessing over the IRR metric? GP’s happily use the subscription line and other games to provide that extra boost.

The list could go on and on…

So why do metrics fail, and more importantly, what can we do about it?

Gaming

Metrics fail because they can be gamed. A simple metric, used in a complex environment with uncontrollable variables, place people in an impossible situation. People begin to get punished for variables outside their control.

At first, people get mad. Then, they get smart and outmaneuver the metric.  

Metrics can’t handle nuance. They can’t handle complexity. They can’t handle the real world.  

Here’s Muller’s take on Gaming:

…gaming the metrics occurs in every realm: in policing; in primary, secondary, and higher education; in medicine; in nonprofit organizations; and, of course, in business. And gaming is only one class of problems that inevitably arise when using performance metrics as the basis of reward or sanction. There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know.

The gaming phenomenon is called Goodhart’s Law.

From The Tyranny of Metrics:

…we have Goodhart’s Law, which states, “Any measure used for control is unreliable.” To put it another way, anything that can be measured and rewarded will be gamed.

Dysfunctional Organizations Love Metrics

Dysfunctional organizations love metrics because it removes the need to think. No more thinking about the ugly realities. Just create beautiful fiction. Look at the metric and pass judgment: good or bad, pass or fail, success or failure. Metrics replace management. Metrics remove judgement, all in the name of “rigorous quantification.”

From The Tyranny of Metrics:

Professionals tend to resent the impositions of goals that may conflict with their vocational ethos and judgment, and thus morale is lowered. Almost inevitably, many people become adept at manipulating performance indicators through a variety of methods, many of which are ultimately dysfunctional for their organizations. They fudge the data or deal only with cases that will improve performance indicators. They fail to report negative instances. In extreme cases, they fabricate the evidence.

Easy Metrics Evaluating The Wrong Thing

Easy metrics are seductive. Easy is fine if it measures what you ultimately desire. Most often, it’s not the case. Many easy metrics are irrelevant.

Here’s Muller’s explanation:

Measuring the most easily measurable. There is a natural human tendency to try to simplify problems by focusing on the most easily measurable elements. But what is most easily measured is rarely what is most important, indeed sometimes not important at all.

Inputs Over Outputs

Don’t treat inputs and outputs as equally valuable. Measuring inputs is easy: hours worked or money spent. But to track what matters, measure the output. Measure what is produced with the hours worked. Measure what is achieved with the money spent.

From The Tyranny of Metrics:

Measuring inputs rather than outcomes. It is often easier to measure the amount spent or the resources injected into a project than the results of the efforts. So organizations measure what they’ve spent, rather than what they produce, or they measure process rather than product.

Complicatedness

Didn’t think your work life was complicated enough? Let’s add more metrics and make it more complicated…

From The Tyranny of Metrics:

The problem is that management’s quest to get a handle on a complex organization often leads to what Yves Morieux and Peter Tollman have dubbed “complicatedness”: the expansion of procedures for reporting and decision-making, requiring ever more coordination bodies, meetings, and report-writing. With all that time spent reporting, meeting, and coordinating, there is little time left for actual doing.

Metrics, like the universe, never stops expanding. Once metric obsession takes hold, everything gets measured. In doing so, life gets complicated. People lose track of what’s important. There’s too much noise and worthless metrics that it becomes a quagmire. It becomes a job to aggregate metrics, let alone interpreting their meaning.

Numbers Over Expertise

Metrics can’t replace expertise. Expertise has flaws, but metrics compound them. The desire for quantification and numerical sophistication doesn’t absolve the need for human judgment.

Here’s the final thought from Jerry Muller:

McNamara’s Pentagon was characterized by what the military strategist Edward Luttwak called “the wholesale substitution of civilian mathematical analysis for military expertise. The new breed of the ‘systems analysts’ introduced new standards of intellectual discipline and greatly improved bookkeeping methods, but also a trained incapacity to understand the most important aspects of military power, which happen to be nonmeasurable.”

Never forget the non-measurable.