by Brent Chapman | Apr 21, 2026 | Uncategorized
Most of what’s been written about blameless post-incident reviews is about managers not blaming engineers, and engineers not blaming each other, because blame shuts down learning. What many miss is that engineers still blame themselves, and the damage is the same.
The most valuable output of a post-incident review isn’t the document, and it isn’t the list of action items. It’s learning: individual learning for the people involved, and organizational learning that outlasts anyone’s tenure. Everything else in the review process serves that goal.
The scene
Somewhere in the middle of a post-incident review, one of the engineers involved in the incident speaks up:
“Yeah, I should have caught that. My bad. I’ll be more careful next time.”
On the surface, this feels like maturity. The engineer isn’t defensive; they’re owning what happened. The team nods, accepts the explanation, and the review moves on.
What actually just happened is that the exploration stopped. “Sam made a mistake” became the story, and all the systemic factors that contributed to the incident went unexamined. The dashboard that was slow to load. The deployment pipeline that didn’t have a canary phase. The runbook that was last updated eighteen months ago. The schedule pressure that made skipping the check seem reasonable in the moment. None of that got talked about, because “I’ll be more careful next time” was accepted as the explanation, when it doesn’t really explain anything.
Self-blame is still blame
Blame shuts down learning. That’s true whether the blame comes from a manager, from a peer, or from self-blame. A satisfying-sounding explanation arrives early in the conversation, everyone accepts it, and the exploration that would have surfaced the actual contributing factors never happens. The document captures a neat narrative. The action items address the narrow issue. The next incident reveals that the narrow issue was a symptom of something deeper that nobody investigated.
“I should have caught that” is a thought-terminating clichรฉ that sounds like insight, but isn’t. It works just as well whether someone else says it or the person says it about themselves.
The heroism problem after the incident
If your company has spent any time thinking about incident response, you’ve probably internalized a version of this principle: individual heroics are a problem during incidents. The lone-wolf engineer who goes heads-down and fixes the outage by personal brilliance and sheer force of will is not the model you want. The incident runs longer than it needed to, nobody else learns anything, and the company ends up dependent on a handful of heroes who burn out or leave. Organizational capability beats individual heroics.
Self-blame is the same pattern, just moved later in time.
The engineer who says “my bad, I’ll be more careful next time” is volunteering to carry the weight of the incident themselves. They’re taking the hit so the team can move on. It feels selfless, even noble. It also deprives the team of the conversation it actually needs to have about the contributing factors behind the incident. The hero absorbs the cost, and the company never learns what it needed to learn.
The response to heroism during incidents is to build structures that don’t require it: defined roles, clear handoffs, explicit delegation, training that spreads capability across the team. The response to heroism after incidents should be the same: don’t let one person carry (and bury) what needs to be shared.
What to do when you see it
Self-blame shows up in a handful of recognizable forms. “My bad, I’ll be more careful.” “I should have known better.” “I take full responsibility for this.” “That was on me.” Whichever variant surfaces, the review needs someone else to gently redirect. Not to argue with the person, and not to wave them off. The work is to pull the conversation from the person’s character back to the situation they were in.
A question that works: “OK, but let’s understand the circumstances. What were you seeing at the time? What information did you have? What made the action seem like the right thing to do?”
This question does several things at once. It signals that the team isn’t satisfied with the self-blame as the explanation. It treats the engineer’s actions as reasonable given what they knew, which is almost always the case. And it opens up the conversation about the circumstances the engineer was working in: the missing information, the ambiguous signals, the process that everyone knew was broken but nobody had fixed. That’s where the contributing factors live. That’s where the learning lives.
Facilitators of post-incident reviews should anticipate self-blame, and be prepared to address it.
The person in the middle
There’s another aspect of this that seldom gets the attention it deserves: the toll on the person absorbing the blame.
The engineer most involved in the incident often arrives at the review already carrying guilt and anxiety; the “my bad” moment is frequently the release of pressure they’ve been carrying for days. Treating it with care, neither accepting the self-blame as the answer nor challenging the person, is part of the job.
The burden doesn’t ease when the review ends. The engineer absorbing the blame carries it for a long time afterward. In healthcare, this is called “second-victim syndrome”: dealing with a patient’s trauma sometimes causes adverse emotional effects for the healthcare provider, as well, making them the “second victim.”
The stakes in software are different, but the experience isn’t as different as it might sound. Engineers lose sleep over incidents. They dread going to work. Some of them leave their jobs. “I should have caught that” sounds like a quick self-assessment; in practice, it’s often a thought the person has been rehearsing for days before the meeting.
A well-run review that explores the systemic factors behind the incident is one of the most effective interventions for this. It reframes the experience for the person involved. They’re no longer “the person who caused the outage.” They’re a participant in a systemic event the company is working to understand and prevent. That reframing matters enormously, and it can’t happen if the review stops at accepting the self-blame as the answer.
Variations to watch for
Individual self-blame has relatives worth watching for. Group self-blame sounds like “we should have caught that”; it diffuses the ownership across the team but stops the conversation the same way. Passive voice (“the change was deployed without adequate testing”) strips the actor from the sentence without removing the judgment. “Human error” without naming the human is blame at one remove, assigned to a generic stand-in. All of these give the review a stopping point that feels satisfying but isn’t actually useful. The redirect is the same in each case: pull the conversation from implicit character judgments back to the situation people were in.
The principle
The next stage of blameless maturity isn’t arriving at a review where nobody points fingers. It’s arriving at a review where the conversation keeps going past “my bad, I’ll be more careful next time” and into the system that made the moment possible.
Fred Hebert has written about a related anti-pattern he calls “superficial blamelessness“: reviews that successfully avoid retribution but still land on individualistic fixes (more training, pay more attention, add supervision) rather than changes to the system. Self-blame is a particularly sneaky version of that pattern. The engineer volunteers the individualistic remediation on themselves, which makes it feel like accountability instead of the shallow fix it is.
Accountability means understanding how your company and its systems actually operate, and making durable changes based on that understanding. One person promising to try harder doesn’t get you there.
I’m writing a book, Incident Management for DevOps and SRE, aimed at helping companies build incident management capability that doesn’t depend on heroics. Sign up for updates at im4ds.com.
If your company needs help with incident management right now, my consulting practice is at greatcircle.com/im.
by Brent Chapman | Apr 14, 2026 | Uncategorized
Ninety minutes into an outage, the person who’s supposed to be running the incident has lost the plot. They’re hunched over a terminal, or deep in a dashboard, or engrossed in a Slack thread debugging the problem alongside the responders. Nobody’s sending status updates. Nobody’s fielding questions from the support team. Nobody’s thinking about whether the response needs more people, or different people, or a completely different approach. The incident commander (IC) has disappeared into the technical work.
When this happens (and it happens constantly), companies tend to treat it as a discipline problem. “The IC got sucked in again.” “We need ICs who can stay above the fray.” As if the solution were stronger willpower.
This is a cognitive problem. Technical troubleshooting and incident coordination require fundamentally incompatible kinds of attention, and doing both at once means neither gets the attention that it needs.
Two rhythms that don’t mix
Technical troubleshooting has a rhythm. It’s heads-down, focused, and sustained, even when you’re working the problem with other responders. You’re following a thread: correlating timestamps, forming a hypothesis, testing it, adjusting, testing again. The work rewards concentration. Interruptions are expensive; every time you break your focus, you lose the mental model you’ve been building and have to reconstruct it. The best troubleshooting happens when you can tune everything else out and just chase the problem.
Incident coordination, the incident commander’s actual job, has the opposite rhythm. It’s heads-up, scanning, bouncing from thing to thing. “Has the networking team checked in?” “Where’s that status update?” “Do we need to loop in the database on-call?” “The VP of engineering just asked for an ETA.” The work requires constant context-switching. Staying focused on any one thread for too long means everything else drifts. If the incident commander spends ten minutes deep in a technical discussion, they’ve missed three stakeholder questions, a newly joined responder has no idea what to work on, and nobody outside the response has heard anything since the incident started.
These two rhythms don’t just coexist poorly, they actively fight each other. The focus required for effective debugging is exactly what makes for ineffective coordination. The constant interruptions that are part and parcel of coordination are exactly what make for ineffective debugging.
The gravitational pull of debugging
When one person tries to do both, the technical work almost always takes over. This isn’t surprising. The technical work is tangible, intellectually engaging, and feels more immediately productive. You’re making progress, finding clues, narrowing the problem. The coordination work is less satisfying in the moment. Writing a status update doesn’t feel like fighting the fire, it feels like paperwork.
So the IC drifts. They open a dashboard “just to check something.” They start a query “just to confirm a hunch.” Twenty minutes later, they’re deep in the investigation and the coordination work has stopped entirely. Nobody told them to stop being the IC, and they didn’t make a conscious decision to stop. They just drifted away from it, because the technical problem was right there and it was interesting and they could help.
The people around them usually won’t say anything, either. The responders are glad to have another strong technical mind on the problem. The stakeholders waiting for updates assume someone is handling it. By the time anyone notices that coordination has stopped, the damage is done: stakeholders are confused, new responders have self-dispatched to random tasks, and nobody has a clear picture of the overall situation.
What to do about it
The fix is structural: don’t expect one person to both lead the response and do technical work within the response.
That expectation often comes from the IC themselves. A strong engineer who’s “just coordinating” can feel like they’re not pulling their weight, especially when they can see exactly what needs to be tried next. If the IC genuinely has unique knowledge that the response needs (they built the failing system, they’ve seen this failure mode before), the right move is for them to hand the IC role to someone else and join the response as a responder. Trying to do both isn’t a good solution.
Think of an orchestra conductor. The conductor doesn’t play an instrument; the orchestra as a whole is their instrument. The moment the conductor picks up a violin, nobody’s conducting. The same thing happens when an IC opens a terminal.
The incident commander doesn’t debug the outage itself; they debug the incident response.
The IC stays in the coordination rhythm: tracking the response, communicating outward, and keeping the big picture in mind so they can make what are often called “sacrifice decisions.” Should we sacrifice the last hour of customer data to roll back to a known good state? Should we keep the storefront down and fix the problem properly, or bring it back up in a degraded state and risk a second outage? Should we notify customers now with incomplete information, or wait until we know more? The technical team can lay out the options, but these tradeoffs cut across teams and affect the business in ways that someone heads-down in a terminal can’t see.
On a small incident (i.e., one involving the IC and just one or two responders), you don’t need any formal role designations among the responders; you just need a clear split between coordinating (by the IC) and troubleshooting (by the responders). On bigger or more complex incidents, where several responders are working the problem, it often makes sense to designate one of them as the tech lead for the response.
The IC and tech lead roles face in opposite directions. The IC faces outward, toward the rest of the company: stakeholders, executives, support teams, other engineering groups that might be affected. The tech lead faces inward, toward the problem: directing the investigation, synthesizing what the responders are finding, making the tactical calls about what to try next. Each one watches a different part of the horizon, and together they cover the full picture.
The IC and tech lead stay in close touch with each other. The tech lead gives the IC a clear, concise summary of where the investigation stands and what the team needs. The IC keeps stakeholders informed, and makes sure the technical team has what it needs to keep moving: resources (including additional responders, if needed), priorities, and decisions on the tradeoffs that aren’t visible from inside the investigation. Together, they let the investigation go deep without the response going dark. Neither one has to switch cognitive modes. Each stays in the rhythm that makes them effective.
When the IC stays in the coordination rhythm, the scene looks different. Stakeholders are getting updates. New responders know what to work on. The responders are heads-down, uninterrupted, chasing the problem. And nobody had to be told to “just try harder.”
I’m writing a book on incident management for engineering teams. If this resonates, visit im4ds.com to follow along.
If your company is working through challenges like this one, I do consulting and training on incident management for engineering companies.
by Brent Chapman | Apr 7, 2026 | Uncategorized
Who was the greatest incident commander of all time? My money is on Gene Kranz.
If you’ve seen the 1995 movie Apollo 13 (and if you haven’t, you should; it’s a gripping tale even though you know the outcome, and along the way it’s a master class in incident command), you’ve seen Ed Harris portray Kranz during one of the most harrowing incidents in the history of human spaceflight. In April 1970, an oxygen tank explosion crippled the Apollo 13 spacecraft roughly 200,000 miles from Earth, turning what was supposed to be the third Apollo landing on the Moon into a desperate fight to bring three astronauts home alive. Kranz was NASA’s lead flight director in Mission Control throughout the multi-day crisis.
What makes Kranz the gold standard for incident command isn’t what he knew. It’s what he did (and more importantly, what he didn’t do).
His job wasn’t to solve the problem
Watch the crisis scenes in the movie closely. Kranz doesn’t try to solve the technical problems himself. He has a room full of brilliant engineers for that. Each of those engineers, in turn, is backed by a “back room” of specialists focused on specific spacecraft systems. Kranz’s job isn’t to solve the problem. His job is to ensure that the problem gets solved, and that’s a subtle but critical difference.
That distinction is fundamental to effective incident command, and many companies get it wrong.
The incident commander role often falls to the most senior engineer by default, with no regard for the practical consequences. The technical work suffers because their best problem-solver keeps getting pulled away to handle logistics and communication, and the overall response suffers because nobody is focused full-time on running it.
What he actually did instead
Kranz was technical enough to understand what his engineers were telling him, and trying to tell each other, but he wasn’t running the calculations himself. So if he wasn’t personally solving the problem, what was he doing instead? His focus was on ensuring that the right people were working the right problems at the right time, not on directly working any of the problems himself.
Watch the movie scenes carefully, and you’ll see a masterclass in incident command:
He organized the response. He set priorities, assigned responsibilities, and made sure every critical task had someone working on it. When the crisis began, he immediately reframed the mission: “From this moment on, we are improvising a new mission: how do we get our people home?”
He demanded clear information. He asked sharp questions and required his team to give him straight answers, not hedged guesses. When the crisis began and readings were contradictory, his instinct was to cut through the speculation: “Let’s work the problem, people. Let’s not make things worse by guessing.”
He made decisions under uncertainty. His engineers disagreed about whether to attempt a direct abort or use the Moon’s gravity to slingshot the crew home. Both options carried enormous risk. Kranz listened to the competing proposals, weighed the tradeoffs, and decided. Then he explained his reasoning so everyone was on the same page.
He kept them looking for possibilities, without being fixated on problems. While his engineers were focused on the individual failures they were trying to address, Kranz stepped back and asked: “What do we have on the spacecraft that’s good?”, challenging them to look for opportunities to use capabilities in unexpected ways to further the mission. When told that a certain system wasn’t designed to do what they needed, he shot back: “I don’t care about what anything was designed to do, I care about what it CAN do.”
He facilitated, not dictated. When engineers brought counter-proposals, he considered them and changed his mind when they were right. When discussion got heated, he reasserted calm: “Let’s hold it down, people.” He allowed debate when it was productive and cut it off when it wasn’t.
He managed the emotional temperature. In military and emergency management circles, this quality is called “command presence.” It doesn’t mean barking orders or being stoic. It means being the steady center of gravity for the response. Responders unconsciously look to the incident commander for cues about how bad things are and whether the situation is under control. If they’re calm and focused, that calmness radiates to the team. If they’re frantic, the team gets frantic too.
Kranz was the calmest person in the room precisely when the room needed calm the most. Not because the situation wasn’t dire, but because he understood that if he lost his composure, everyone else would too.
“Failure is not an option”
Kranz’s famous line from the movie has become a clichรฉ, but it’s worth revisiting. Watch the scene again. It’s not bravado. It’s a statement of intent from someone who has assessed the situation, accepted the constraints, and decided that the team is going to find a way through. He doesn’t pretend the situation isn’t terrible. He tells his team exactly how bad it is, exactly what they need to do, and then sets the expectation that they will find a way to do it.
That’s what good incident command looks like. Not magical thinking, not denial, not heroics. Clear-eyed assessment, clear decisions, and relentless follow-through.
The lesson for your incident response
Your incidents probably don’t involve life-or-death stakes while the whole world watches anxiously. But the model is the same.
The incident commander’s job is to ensure that the problem gets solved, not to solve the problem themselves. That means organizing the response, making decisions, tracking what’s been tried and what hasn’t, communicating with stakeholders, and being the steady center of gravity so everyone else can do their best work.
Many companies default to making their most senior engineer the incident commander. It seems reasonable: they have the most authority, the most experience, the most knowledge.
But your incident commander doesn’t need to be your top engineer. They need to be the person who can stay calm, make decisions with incomplete information, facilitate a room full of smart people who disagree, and ensure that nothing falls through the cracks while everyone else focuses on the technical work.
Gene Kranz showed what incident command looks like when it’s done right. More than half a century later, he’s still the gold standard to aspire to.
I’m writing a book about incident management for software engineering teams, drawing on lessons from both the tech industry and public safety (and movies about NASA!). If you’d like to hear when it’s available, visit im4ds.com.
If your company needs help with incident management right now, my consulting practice is GreatCircle.com/im.
Recent Comments