People sometimes ask me, “How do I reopen a closed incident?” The short answer is that you shouldn’t. Open a new incident instead.
There are situations where reopening a closed incident might seem like the right thing to do. Perhaps the incident was closed prematurely, before the problem was fully understood and truly resolved. Or perhaps the problem was understood fully, but occurred again before you could implement a permanent preventative measure. In any case, it’s very tempting to say, “Hey, we already had an incident for this; let’s just reopen it and keep going!” but this can cause all sorts of problems.
The first problem with reopening a closed incident is that the folks who responded to the original incident have returned to their other projects, signed off for the day, or whatever. You have to re-assemble your team, just like you did in the first place at the start of the prior incident.
Second, reopening an incident leads to confusion. “Hmm, we dealt with that already; this new alert must just be a late duplicate, so I’ll ignore it.” Responders and observers aren’t clear on who is re-responding, who the Incident Commander is now, etc.
Third, closing an incident usually kicks off a variety of post-incident activities that have to be reversed to reopen the incident: closure of the incident is reported to executives, customer support teams, and other interested parties; someone is assigned to schedule and conduct the post-incident review; skeletal incident review (postmortem) docs are automatically created from templates; etc. To reopen an incident, all that work needs to be rolled back or reversed, and then repeated when the reopened incident is eventually closed again, leading to further chaos and confusion.
The best solution is to open a new incident. In doing so, refer to the prior incident; this may help close the second incident more quickly, given what you learned the first time around. When you close the second incident, it’s OK to do a single joint incident review that covers both incidents; it should also explore why a second incident was needed.
Are there any circumstances under which I would reopen a closed incident? Maybe, but only if I realized within minutes of closing that closing was a mistake, and none of the post-incident activities had kicked off yet. Once more than a few minutes have passed and the incident responders have dispersed, or any of the post-incident activities have begun, it’s too late; at that point, attempting to re-open causes more confusion than simply opening a new incident.
—
Outages and other IT emergencies are expensive in many different ways, including lost sales, lost productivity, damaged reputation, and damaged morale. It’s essential to be prepared, and to learn from each incident so that you’re better prepared for next time. I can guide your organization to develop these critical incident management capabilities. Contact me to learn more, or to schedule a free tech talk on incident management for your team or organization.
Recent Comments