Ninety minutes into an outage, the person who’s supposed to be running the incident has lost the plot. They’re hunched over a terminal, or deep in a dashboard, or engrossed in a Slack thread debugging the problem alongside the responders. Nobody’s sending status updates. Nobody’s fielding questions from the support team. Nobody’s thinking about whether the response needs more people, or different people, or a completely different approach. The incident commander (IC) has disappeared into the technical work.

When this happens (and it happens constantly), companies tend to treat it as a discipline problem. “The IC got sucked in again.” “We need ICs who can stay above the fray.” As if the solution were stronger willpower.

This is a cognitive problem. Technical troubleshooting and incident coordination require fundamentally incompatible kinds of attention, and doing both at once means neither gets the attention that it needs.

Two rhythms that don’t mix

Technical troubleshooting has a rhythm. It’s heads-down, focused, and sustained, even when you’re working the problem with other responders. You’re following a thread: correlating timestamps, forming a hypothesis, testing it, adjusting, testing again. The work rewards concentration. Interruptions are expensive; every time you break your focus, you lose the mental model you’ve been building and have to reconstruct it. The best troubleshooting happens when you can tune everything else out and just chase the problem.

Incident coordination, the incident commander’s actual job, has the opposite rhythm. It’s heads-up, scanning, bouncing from thing to thing. “Has the networking team checked in?” “Where’s that status update?” “Do we need to loop in the database on-call?” “The VP of engineering just asked for an ETA.” The work requires constant context-switching. Staying focused on any one thread for too long means everything else drifts. If the incident commander spends ten minutes deep in a technical discussion, they’ve missed three stakeholder questions, a newly joined responder has no idea what to work on, and nobody outside the response has heard anything since the incident started.

These two rhythms don’t just coexist poorly, they actively fight each other. The focus required for effective debugging is exactly what makes for ineffective coordination. The constant interruptions that are part and parcel of coordination are exactly what make for ineffective debugging.

The gravitational pull of debugging

When one person tries to do both, the technical work almost always takes over. This isn’t surprising. The technical work is tangible, intellectually engaging, and feels more immediately productive. You’re making progress, finding clues, narrowing the problem. The coordination work is less satisfying in the moment. Writing a status update doesn’t feel like fighting the fire, it feels like paperwork.

So the IC drifts. They open a dashboard “just to check something.” They start a query “just to confirm a hunch.” Twenty minutes later, they’re deep in the investigation and the coordination work has stopped entirely. Nobody told them to stop being the IC, and they didn’t make a conscious decision to stop. They just drifted away from it, because the technical problem was right there and it was interesting and they could help.

The people around them usually won’t say anything, either. The responders are glad to have another strong technical mind on the problem. The stakeholders waiting for updates assume someone is handling it. By the time anyone notices that coordination has stopped, the damage is done: stakeholders are confused, new responders have self-dispatched to random tasks, and nobody has a clear picture of the overall situation.

What to do about it

The fix is structural: don’t expect one person to both lead the response and do technical work within the response.

That expectation often comes from the IC themselves. A strong engineer who’s “just coordinating” can feel like they’re not pulling their weight, especially when they can see exactly what needs to be tried next. If the IC genuinely has unique knowledge that the response needs (they built the failing system, they’ve seen this failure mode before), the right move is for them to hand the IC role to someone else and join the response as a responder. Trying to do both isn’t a good solution.

Think of an orchestra conductor. The conductor doesn’t play an instrument; the orchestra as a whole is their instrument. The moment the conductor picks up a violin, nobody’s conducting. The same thing happens when an IC opens a terminal.

The incident commander doesn’t debug the outage itself; they debug the incident response.

The IC stays in the coordination rhythm: tracking the response, communicating outward, and keeping the big picture in mind so they can make what are often called “sacrifice decisions.” Should we sacrifice the last hour of customer data to roll back to a known good state? Should we keep the storefront down and fix the problem properly, or bring it back up in a degraded state and risk a second outage? Should we notify customers now with incomplete information, or wait until we know more? The technical team can lay out the options, but these tradeoffs cut across teams and affect the business in ways that someone heads-down in a terminal can’t see.

On a small incident (i.e., one involving the IC and just one or two responders), you don’t need any formal role designations among the responders; you just need a clear split between coordinating (by the IC) and troubleshooting (by the responders). On bigger or more complex incidents, where several responders are working the problem, it often makes sense to designate one of them as the tech lead for the response.

The IC and tech lead roles face in opposite directions. The IC faces outward, toward the rest of the company: stakeholders, executives, support teams, other engineering groups that might be affected. The tech lead faces inward, toward the problem: directing the investigation, synthesizing what the responders are finding, making the tactical calls about what to try next. Each one watches a different part of the horizon, and together they cover the full picture.

The IC and tech lead stay in close touch with each other. The tech lead gives the IC a clear, concise summary of where the investigation stands and what the team needs. The IC keeps stakeholders informed, and makes sure the technical team has what it needs to keep moving: resources (including additional responders, if needed), priorities, and decisions on the tradeoffs that aren’t visible from inside the investigation. Together, they let the investigation go deep without the response going dark. Neither one has to switch cognitive modes. Each stays in the rhythm that makes them effective.

When the IC stays in the coordination rhythm, the scene looks different. Stakeholders are getting updates. New responders know what to work on. The responders are heads-down, uninterrupted, chasing the problem. And nobody had to be told to “just try harder.”

---

I’m writing a book on incident management for engineering teams. If this resonates, visit im4ds.com to follow along.

If your company is working through challenges like this one, I do consulting and training on incident management for engineering companies.