Mitigations, fixes, and root causes

Considering a different framing for economic and political problems

In my professional life, I'm familiar with the distinction between mitigating a problem and fixing a problem. When there is a crisis with a system, the focus should be on mitigation: finding a way to restore normal function, even if it’s just cobbled together or temporary. A classic everyday example of mitigation is room freshener: a spray of perfume can help cover up or distract from a problematic odor, but whatever is causing the odor (garbage not taken out? a toilet that needs cleaning? spoiled food in the pantry?) remains, and is in no way fixed by the spray. Of course, sometimes an obvious problem is quickly diagnosed and fixed, so no separate mitigation is required; but, in general, a real (permanent) fix is slower than one or more quick (temporary) mitigation actions.

Engineers are often keen to do things right, which prompts them to disparage quick hacky solutions. Nevertheless, a fast mitigation is genuinely important for the users of the system. As long as proper (seeming?) function has been restored rapidly, they don’t really care about how cheesy the mitigation might have been.

At the same time, an engineer’s concern with a correct fix is well placed: it’s equally erroneous – and in some ways more harmful – to treat a quick hack as the end of the problem and move on. In the case of the problematic odor, it will likely recur until it’s genuinely fixed. In the case of software systems, the quick patches and hacky approaches create technical debt: as with financial debt, there is an ongoing incremental “interest” cost in the form of new kinds of failures, especially as “temporary” fixes start to interact in unanticipated ways, since each was only intended to mitigate a single concern. These kinds of interest costs can only be eliminated by paying down the debt (putting in real, permanent fixes).

Root cause analysis

I'm also well acquainted with the methodology involved in finding root causes for a particular incident. The underlying motivation is that failures typically have complex causes, with multiple contributing elements. Simple theories may work adequately for mitigation, without being completely correct: often, a partial understanding of the problem is enough to patch together a partial solution. But getting the fix right requires more attention.

Doing root cause analysis well requires an attitude of impartial investigation. The goal is to avoid blame but acknowledge responsibility, which can be a tricky balance. One important element is a summary of all that is known about what went wrong – the timing is often particularly important – as well as what was learned from the incident, and steps to be taken to avoid similar problems in the future.

Crucially, the process doesn’t end with the construction of that summary. Instead, the resulting document then undergoes a cross-functional review, typically a meeting involving both technical experts and senior management. This review helps avoid blame-shifting or scapegoating: it’s hard to blame someone else if they’re in the meeting and in a position to contradict you. This review also helps ensure that necessary changes happen: the relevant leaders are present and understand the implications of the problem. Quite often, the conclusions from the wider review are different from what would have been concluded from the summary alone.

Non-technical mitigations

It recently occurred to me that some of the same concepts are applicable in economic and political contexts: more specifically, much of our current politics seems to fall into the category of poorly motivated mitigations, combined with a lack of interest in either fixes or root causes.

I recently read a description of a tech company employee’s charitable project: building some gadget to help medically underserved people in India. The point of the article was that the employee was a noble person, the technology was interesting and useful, and that the employee and his efforts were making the world a better place. All well and good.

But for some reason, on this occasion, I found myself asking about root causes: why are those people in India medically underserved? Why do we accept that the medical care – in this case, eye care – that is routinely available in the United States or other western industrial countries simply cannot be made available in India? Why are we praising some hacky scheme devised by a tech employee in their spare time?

This is not a particularly important example, and my point is not to solve it specifically. You can likely find many similar examples of mitigation vs. fixes, if you go looking for them. And none of this is to fault the employee’s effort at mitigation. Even thoughtful engineers who want to do a technically sound fix will prefer a mitigation to an unmitigated problem, if those are the only two choices.

Could we do better?

The larger question here is, why is there so little thinking about this distinction, and why are there so many political and economic situations in which it feels like we just patch and move on?

One problem, of course, is that we seem to have serious divisions about the basics of what constitutes success vs. failure, and what our collective goals should be.

I’ve written previously about some related topics: poison ivy politics, what-about-ism, politics in an attention economy, bogus arguments, subtle and unsubtle ways of lying, and developing a scientific attitude.

My professional experience of root cause analysis benefits from the deliberate selection of employees who are aligned with what the company’s investors and management want to accomplish. For better or worse, a democracy doesn’t function that way. Although there can be a temptation to filter out the people who disagree with us, in principle all voters are equally relevant to these processes. Accordingly, one possible explanation of our political situation is that anything beyond the simplest mitigation is just too hard.

I suspect another part of the answer is that if we do examine root causes, we probably won't like the answer. Certainly, the early days of using root cause analysis in engineering organizations can be hard. The meetings tend to turn up embarrassing situations, poor judgment, and incompetence. The challenge for all the participants is to treat those aspects as by-products, rather than the focus. For example, when something is embarrassing, it’s important to acknowledge it… but neither hide it, nor spotlight it. The embarrassing situation is simply a historic fact about something that we’d prefer not to repeat. The focus should be on what we can change to avoid repetition of the problem.

At some level, it seems likely that people with any degree of intelligence and curiosity in the wealthy nations of the world know that they are complicit in creating these problems, or at least in maintaining the status quo that creates such problems. But there seems to be a lot of what I might characterize as either willful ignorance, or perhaps an unwillingness to think through implications. (I recognize that I am myself guilty of these kinds of avoidant behaviors, and probably more often than I realize).

I don’t know what the political equivalent of root cause analysis would be, or how the process should work. But I increasingly think that some form of honest and searching approach to our problems is necessary to pay down our collective “political debt.”