To minimize incident impact, focus first on mitigation, not resolution

During an incident, it’s important to focus first on mitigation, not resolution. Mitigation is “stopping the bleeding”; it’s stopping customers from being impacted by the incident (or at least stopping any more customers from being impacted by it, beyond those who already have been). It’s the critical first goal in the incident process.

Once you achieve mitigation, you may still have an emergency on your hands, and you should continue to treat it as such. You might reach mitigation without yet fully understanding what went wrong and what needs fixing. You may still need to recover the system to a stable state, find and repair the bugs or conditions that triggered the incident, restore lost or corrupted data, etc. 

Conversely, resolution is when you no longer need to treat the situation as an emergency. You’ve done all the emergency work, and there may still be follow-up work to do, but the follow-up will be handled through your normal non-emergency processes (support tickets, bug reports, Agile/DevOps processes, etc.).

Mitigation is the critical first goal; once we’ve achieved mitigation, we switch our sights to resolution. On many incidents (especially small or straightforward ones), mitigation and resolution are equivalent; when you’ve mitigated it, you’ve also reached the point where it makes sense to call it “resolved.” But not every incident is that straightforward, so it’s helpful to keep “mitigation” and “resolution” in mind as separate concepts.

So, how can you focus first on mitigation? I have two suggestions: develop and use generic mitigations, and beware of the temptation to fix-forward.

Develop and use generic mitigations

Generic mitigations are actions that your incident responders can take to “stop the bleeding” even before they fully understand the underlying causes of the incident. For example, if the impact seemed to start just after releasing new code, you roll back to the prior release; if the servers seem to be overloaded, you spin up more servers to increase capacity; or if requests from a particular user seem to be crashing the server, you temporarily block requests from that user. In each of these cases, you’re taking immediate steps to reduce or eliminate the impact of the incident before you really understand what is causing it (i.e., before you find the bug in the new release, before you determine what has caused the load to increase, or before you figure out precisely why that user’s requests are causing the system to crash). 

By minimizing the customer impact of incidents, generic mitigations buy you time to do the more thorough troubleshooting that’s needed to thoroughly understand what went wrong and devise a proper fix for it. There are various types of generic mitigations, but they all depend on building certain capabilities into your systems in advance, so they’re available when you need them. To learn more about generic mitigations, I recommend the excellent article Generic mitigations: A philosophy of duct-tape outage resolution by Jennifer Mace of Google SRE.

Beware the temptation to fix forward

During incidents, it’s often very tempting to “fix forward”; that is, to skip over mitigation in favor of fixing what you believe is the underlying problem at the heart of the incident. That’s OK to consider, as long as you keep in mind two things: 

• It’s software, so it always takes longer to develop, test, and deploy than you think it will.
• It’s software, so it sometimes has bugs (or “unexpected features”) and makes things worse.

It’s all too easy and all too common to find yourself in a situation where you’re waiting and waiting and waiting for the “almost ready now” fix, while the problem keeps impacting customers. Or in a situation where the eventual “fix” has to be followed by another “fix” to address problems in the first fix. Or both. 

Because of these uncertainties and our tendency to over-estimate how easily/quickly/well a fix-forward solution will work, when I’m acting as Incident Commander, I have a strong bias towards a strategy of “mitigate first, then fix”, where “mitigate” usually means applying one of the generic mitigation strategies discussed above. I can be convinced to “fix-forward” if the relevant subject matter experts are really confident in their fix-forward and can make a compelling argument about why that’s going to be better and result in less customer impact than mitigate-then-fix, but they have to make a pretty strong case to convince me.

—

IT incidents can have significant consequences for businesses, affecting both customers and employees. With customers, incidents can lead to lost revenue, missed sales opportunities, and damage to your reputation. With employees, they can result in decreased productivity, reduced morale, and increased turnover. To mitigate these risks, it is essential to be proactive, prepare for future incidents, and learn from every incident. As an expert incident management consultant, advisor, and coach, I can help your organization develop these critical skills and avoid costly mistakes. Please don’t hesitate to contact me to explore how I can help your team.