The Numbers Matter More Than We Think
The Prevalence of Metric Fixation
Jerry Z. Muller, Professor of History at the Catholic University of America in Washington D.C., delivered a lecture hosted by Notre Dame Institute for Advanced Study entitled, “Our Metric Epidemic: Diagnosis and Prognosis.”
His book, The Tyranny of Metrics, was published last winter. He was led to this topic by a combination of his long-standing interests in capitalism, culture, and public policy. But his experience as chair of his department ultimately drove him to take up the topic of performance metrics.
Over time, he found there were “more and more requests for more and more data,” by the administration. He further explored where the origin of the request was coming from. This led him to the Middle States Commission on Higher Education, the regional membership association that accredits CUA.
Yet he did more research and found that the impetus for all these data requests ultimately came from the Department of Education. This was during the time of the first George W. Bush administration when Margaret Spellings was involved with educational policy and the No Child Left Behind Act. Under the Act, Students from 3rd to 8th grade took standardized tests, and if they underperformed, schools were penalized.
Muller continued to trace back this cultural mentality. In government, it was called “new public management.” If one traces it back further, one finds it was omnipresent in the private sector. And when one traced it back even further, one found “it actually didn’t originate from the businesses, but from business schools and business consultants.”
“I saw there was this broad cultural pattern in contemporary organization,” said Muller.
Professor Brad Gregory, Director of the Notre Dame Institute for Advanced Study, earlier noted that the book, The Tyranny of Metrics, is not about the evils of measuring human performance, rewarding achievement, or transparency as such. “It’s about a perversion of those I’ve called ‘metric fixation’,” Muller noted.
“Metrics, the measurement, is often useful when it’s combined with judgement based upon experience.”
Metric fixation is based on three beliefs that seem reasonable at first but turn out to be unreasonable. They are the following:
- “What gets measured gets done”
- People respond to incentives
- Professionals at institutions become more accountable to the public if they make these standardized measure of performance public
Metric fixation often fails. Professor Muller pointed to a New York Times article published on January 1, 2018, entitled, “At Veterans Hospital in Oregon, a Push for Better Ratings Puts Patients at Risk, Doctors Say.”
The article reveals the characteristic hazards of metric fixation. It revolves around the story of Roseburg Veteran Affairs Hospital in Oregon. They found case selection bias being used in regard to admitting patients to the hospital. Low-risk patients were admitted and high-risk patients were not, in order to help ratings.
Moreover, some patients who may have benefitted from a surgery did not receive one in case they died within 30 days of discharge, as hospitals could be penalized for that, which hurts their ratings.
There was a high turnover of primary care doctors at the VA hospital, indicating goal diversion which occurs in such cases: when metrics become the standard, the care for patients no longer remains the focus. Goal diversion leads to demoralization of staff.
Here are six effects of metric fixation:
- Diversion by the tension of what gets measured and rewarded
- It discourages innovation, initiative, and risk-taking
- Encourages gaming (manipulating measures to meet metric targets)
- Promotes short-termism
- Measuring takes time, energy, and resources
“Metric fixation rose in part to limit the effect of bias,” said Muller, “… Standardized measurement can be useful.”
Metric fixation has become stifling and counterproductive. One should keep in mind that measurement requires judgement based on experience.
Muller then discussed the “ideology of managerialism.”
“It’s based on the notion that management is not a matter of skill or judgement based upon experience or immersion in activity being managed, or doing something for a long time, but management is a set of tools, mostly numerical tools, that one can learn and apply to another organization.”
Due to metric fixation, managers, then, call in consultants to assist them on the job. Other effects of managerialism include removing responsibility and diverting control to those more removed from it.
“A lot of metric fixation is based on a radically simplistic and truncated overly materialistic of motivation. Really, that we are all Pavlovian dogs.”
Related to that is a kind of scientism where people can be successfully measured and manipulated, Muller noted.
And so, what seems likes a banal topic is actually linked to broader issues in our culture.
Bea Cuasay is a sophomore studying Philosophy and Humanistic Studies. She is an aspiring Loeb Classical Library aficionado. If you share a love of Classics, email her at email@example.com.