Most of the time, simultaneous incidents are no big deal. Any company with enough engineers and enough services will have multiple incidents open at the same time; when I led incident management at Slack, we had a few thousand engineers, and half a dozen concurrent incidents was not unusual for a typical weekday. Each had its own incident commander (IC), its own responders, its own channel, and they proceeded independently without anyone needing to think about the others.

Any conflicts that emerged over priority or resources could usually be worked out among the incident commanders of the individual incidents. The most obvious approaches are straightforward: a higher-severity incident takes priority over a lower-severity one, or a customer-impacting incident takes priority over an internal-only one. When only two incidents are in contention, the two ICs can almost always negotiate a decision between themselves.

Sometimes, though, negotiation among the ICs couldn’t produce a timely solution, particularly when more than two incidents were involved.

Consider a large SaaS platform with three active incidents, all with fixes ready, all needing the same deploy pipeline. Incident A is a Sev-1: the customer-facing API is returning errors for a segment of users. Incident B is also a Sev-1: a partner-facing integration is down. Incident C is a Sev-2: a background data pipeline is falling further behind, and if it isn’t addressed in the next few hours, the backlog will cascade into customer-impacting failures worse than A or B. The severity heuristic doesn’t break the tie between A and B. The customer-impact heuristic doesn’t help either: A and B are both customer-impacting in different ways, and C will be soon. With two incidents, the two ICs can talk it through. With three, each IC is advocating for their own incident, and the tradeoffs involve factors that no individual IC has visibility into: the contractual implications of A, the partner relationship at stake in B, the rate at which C’s backlog is growing. Somebody with a cross-incident perspective needs to decide the sequencing.

When I was at Slack, this kind of contention came up about twice a year. The trigger was usually the deploy pipeline for the monolith. Normally, incidents deployed fixes alongside each other and routine code pushes without any issues. But occasionally, an incident’s fix was risky enough to need exclusive use of the pipeline: a staged rollout that could take a couple of hours, where we didn’t want to bypass the staging except in a dire emergency, and didn’t want to bundle high-risk fixes for multiple incidents into the same deploy because rolling back one fix mid-deploy could derail the other. When multiple incidents each wanted exclusive use of the pipeline at the same time, the ICs were stuck.

Three options when incidents collide

When simultaneous incidents start interfering with each other, there are three options.

Leave them separate. If the interference is minor and manageable (one incident can wait an hour for the deploy pipeline, or the resource contention resolves with a quick conversation between the ICs), there may be nothing to do beyond agreeing on a path forward. Not every collision warrants escalation.

Combine them. When investigation reveals that two “separate” incidents share a common underlying cause, consider merging them into a single response. What looked like independent problems turns out to be different symptoms of the same failure, and maintaining separate responses would mean duplicating effort and coordination. But don’t be in too big a hurry to merge: concurrent incidents are common, and correlated incidents are much less common. Merging is also hard to undo; if you realize mid-response that the incidents weren’t actually related, unmerging is about as messy as reopening a closed incident (pro tip: don’t try, open a new incident instead).

Stand up a coordination layer. When multiple active incidents are genuinely distinct (different systems, different expertise needed), but they’re competing for the same scarce resources, and the individual ICs can’t resolve the contention among themselves, somebody needs to make the cross-incident prioritization calls. Keep the incidents separate but add a coordinator above them to make priority, policy, and resource allocation decisions. In the Incident Command System, this coordination layer is called Area Command.

Area Command

To see why the Area Command pattern is needed, consider a tornado outbreak that drops multiple tornadoes across a metro area in quick succession. A school gym hosting a basketball game has partially collapsed with people trapped inside. A residential neighborhood has been leveled, with reports of injuries throughout. A gas line rupture in an industrial area has started a fire that’s threatening an adjacent apartment complex. Each of these is a separate incident with its own IC and responders, but there aren’t enough responders to go around; all three incidents need the same scarce resources.

Heavy-rescue teams are needed at the school gym to reach trapped spectators, but also in the residential neighborhood where people are buried in collapsed houses. Fire engines are needed at the gas fire to keep it from reaching the apartment complex, but the residential neighborhood also has secondary fires breaking out. All three incidents need ambulances. None of the individual incident commanders is in a position to make the tradeoffs across incidents; each one is rightly focused on their own scene. Somebody above them needs to make the hard calls: which scene gets the heavy-rescue teams first? How many of the fire engines go to the industrial fire and how many get sent to the neighborhood? That’s Area Command.

The structure is deliberately lean. The Area Commander doesn’t manage any incident directly. They allocate scarce resources across incidents, set relative priorities, and communicate the aggregate picture to elected officials. Each incident still has its own IC running its own response; Area Command coordinates between them, not within them.

What this looked like at Slack

At Slack, we adapted Area Command for situations where multiple incidents collided and needed coordination above the level of the individual ICs. The Area Command was itself an incident, with its own incident number, its own channel, and its own IC (the “Area Commander,” typically myself or another senior member of the incident management program). The Area Commander would interface with the IC from each active incident (or the IC would name a liaison) and focus on the questions that no individual IC could answer alone: which incident should get priority for deploying fixes, how to consolidate conversations with executive leadership, and whether a scarce resource should stay on one incident or move to another.

These are sacrifice decisions at the company-wide level: deliberately accepting a worse outcome on one incident to get a better outcome on another. The Area Commander would frame the decision for executives when needed, then work with the individual incidents to implement whatever was decided.

What made this work was keeping it lean and temporary. There was no Tech Lead at the Area Command level, because the technical work was happening within the individual incidents, not at the coordination level. If an IC designated a liaison to Area Command rather than handling that themselves, it was never the Tech Lead. The TL’s attention stayed on their incident’s technical workstreams.

We activated Area Command roughly twice a year, typically only for about an hour. It spun up when incidents started tripping over each other, and spun down as soon as the contention was resolved (often well before the incidents themselves were resolved). It’s not a standing organization; it’s a coordination pattern you reach for when you need it and put away when you don’t.

A rare but high-stakes situation

Most companies won’t need Area Command often. But the situations where it’s needed are precisely the situations where you don’t want to be figuring it out from scratch.

The pattern itself is simple: keep the individual incidents running independently, stand up a lightweight coordinator above them, and give that coordinator the authority and information to make the tradeoff calls that no individual IC can make. It doesn’t require elaborate tooling or training. It requires someone senior enough to make prioritization and resource allocation decisions, a channel for the coordination conversation, and a line of communication with each active incident.

The harder part is recognizing when simultaneous incidents have crossed from “running at the same time” into “interfering with each other.” Most companies stay in the first category most of the time. But any company with enough services and enough engineers will eventually hit the second, and having a name for the coordination pattern (and having thought through how it would work) makes the difference between a structured response and several ICs each lobbying for their own incident with no one positioned to make the call.

---

Area Command is one of the patterns I cover in my forthcoming book, “Incident Management for DevOps and SRE.” If you want to know when it’s available, sign up at im4ds.com.

If your company needs help building or improving your incident management practice, that’s the focus of my consulting at Great Circle.