by Brent Chapman | Jul 28, 2026 | Uncategorized
Is it coddling when an on-call engineer takes the next morning off to recover after handling a production incident at 3 a.m., or is it a smart company managing a reliability risk?
Here’s what that night actually looks like. The engineer gets paged at 3 a.m., then spends two hours diagnosing the problem, coordinating with fellow responders, and restoring service. By 5 a.m., the incident is resolved and they get back to bed, but it takes them a while to settle down and get back to sleep.
Four hours later, they’re at standup. That afternoon, they’re in a planning meeting. That night, they’re still primary on the pager.
This is the default at most companies. Nobody made a deliberate decision that it should work this way; it’s just what happens when there’s no explicit policy for post-incident recovery. And it carries more risk than most leaders realize.
Incident response is more fatiguing than regular work
Responding to an incident isn’t like a normal day of developing features and chasing bug reports. The cognitive demands are qualitatively different: rapid context-switching under time pressure, high-stakes decisions with incomplete information, coordinating across multiple people and systems, all while knowing that users are affected and stakeholders are watching. And there’s a physiological dimension that regular engineering work rarely triggers: adrenaline. Incident response activates the body’s stress response in a way that writing code or reviewing a design doesn’t. That heightened state feels productive in the moment, but it depletes reserves fast, and the crash afterward is steeper than the apparent effort would justify.
This, incidentally, is one of the reasons that training and practice matter so much. Responders who’ve rehearsed the process and trust the framework around them experience a less intense stress response when real incidents hit. Turning incident response into a “routine emergency” doesn’t just improve efficiency; it reduces the physiological toll.
An engineer who’s been actively responding for a few hours isn’t just tired in the way that a long day makes you tired. They’re measurably less effective at exactly the skills incident response demands: integrating new information, evaluating competing hypotheses, making decisions under ambiguity, and recognizing when a current approach isn’t working.
The degradation is predictable
Responder fatigue follows a recognizable pattern. As it sets in, people stop processing new information as effectively. They agonize over decisions they’d normally make quickly, or they stop making decisions altogether. They develop tunnel vision, fixating on the one theory they’re already pursuing instead of stepping back to consider alternatives. They fall into ruts, essentially pursuing “Plan A, again, with more feeling this time” instead of asking whether Plan A is still the right plan. They get less creative, more rigid, and more prone to mistakes.
Fire departments study this, because it’s exactly the scenario firefighters face: interrupted sleep from overnight emergency calls, then back on duty the next day. The research consistently shows that the kind of fragmented, insufficient sleep they get around overnight calls degrades next-day cognitive performance to levels comparable to having had a couple of drinks. That’s why a growing number of fire departments have reconsidered their traditional 48-hour shifts; the performance degradation on day two is bad enough that departments are restructuring around it.
Self-reporting isn’t enough
The insidious part is that fatigue undermines exactly the capacity you need to recognize it. A fatigued responder genuinely believes they’re performing normally. That’s simply how fatigue works. “I’m fine, I can keep going” isn’t evidence of fitness. It’s one of the symptoms.
This is the critical organizational point. If a company leaves fatigue management to individual judgment (“take it easy if you need to”), it’s built a system that depends on impaired people accurately assessing their own impairment.
Aviation learned this the hard way. The FAA doesn’t ask pilots whether they feel too tired to fly. It sets hard limits on duty time and required rest periods, because decades of accident investigation proved that self-assessment under fatigue is unreliable. Pilots who’d been awake for 20 hours consistently reported feeling capable. The data said otherwise. If you’ve ever had a flight delayed while the airline sought a new crew because the original crew had “timed out,” you’ve seen these rules in action.
The tech industry hasn’t had its equivalent reckoning yet, but the same cognitive science applies. An engineer who handled a two-hour incident at 3 a.m. and says they’re fine at 9 a.m. may well believe it. That doesn’t mean they’re right, and building your next day around that assumption is a gamble most companies don’t realize they’re taking.
What active fatigue management looks like
Companies that take responder fatigue seriously don’t rely on individual heroism or self-assessment. They build a few specific practices into their incident management capability.
Explicit rest expectations. Not “take it easy if you need to,” but clear guidelines: an engineer who responds to a significant incident overnight is expected to start late or take the morning off, depending on duration and severity. The default is rest; working the next morning is the exception that requires a conscious choice, not the other way around.
The incident commander (IC) monitors for fatigue. During extended incidents, it’s the IC’s responsibility to watch for fatigue signals in responders: slowed decision-making, tunnel vision, repeated questions, irritability, loss of situational awareness. This is the same responsibility a fire officer has for monitoring crew fatigue on a fireground. A fatigued responder who stays on the line isn’t being dedicated; they’re becoming a risk to the response, their teammates, and themselves. On incidents that stretch beyond a few hours, this includes planning responder reliefs early rather than waiting for someone to admit they’re spent.
Promote the backup. After a significant overnight incident, consider moving the backup on-call engineer to primary for the next 12 to 24 hours. The person who spent two hours at 3 a.m. restoring service is not the person you want as your first line of defense if something else breaks that afternoon.
Rethink shift length. Most teams seem to default to week-long on-call shifts with several weeks between shifts, but there’s a strong case for shorter, more frequent shifts. The same logic driving fire departments away from 48-hour shifts applies: shorter shifts mean less accumulated fatigue per shift, even if each person’s total on-call hours per quarter are similar. Shift design is a fatigue management decision, whether your company treats it as one or not.
Some incident management platforms are starting to build this awareness into their tooling. incident.io, for example, detects overnight pages and proactively asks the responder the next day whether they’d like someone to cover their next shift. That’s the right instinct: making fatigue management a system-level concern rather than leaving it to the judgment of the person who’s least equipped to assess it.
It’s a reliability decision
Most companies aren’t actively choosing to ignore a fatigue problem. Rather, they have a fatigue problem that they haven’t noticed yet, because nobody has framed it as an operational risk. When a leader says “we trust our engineers to manage their own energy,” what they’re actually saying is: we have no organizational mechanism for ensuring that the people responding to our next incident are cognitively fit to do so.
Respecting fatigue isn’t coddling. It’s protecting the quality of everything your engineers do the next day, including the next incident response.
by Brent Chapman | Jul 21, 2026 | Uncategorized
It’s 3 a.m. in California, where most of the dev team are still snug in their beds. The auth system has started rejecting valid credentials. Early bird East Coast customers are already trying (and failing) to log in for the day, and thousands of users in Europe have already given up and gone elsewhere. In a couple of hours, the West Coast will be waking up too. A brilliant engineer swoops in and saves the day. She has legendary debugging skills and a deep understanding of the auth system, and she puts together a fix in forty minutes that would have taken anyone else hours to even diagnose. Later that morning, leadership is sending thank-you messages in the all-hands channel. Her VP awards her a small spot bonus, and her manager reminds her to include it in the next performance review cycle.
What doesn’t usually happen is anyone asking: what if she hadn’t been there? Because that heroic save, for all the heartfelt celebration around it, was actually a near miss from a systemic point of view.
Near misses look like successes
In aviation and other safety-critical fields, it’s widely accepted that a near miss is an unparalleled opportunity to learn and deserves the same investigation as an actual failure. The reasoning is straightforward: a near miss reveals the same systemic vulnerabilities that a failure does. The only difference between a near miss and a disaster is that the outcome happened to be good this time, often because of luck, timing, or the presence of one specific person.
A heroic incident response is a similar opportunity. The system nearly failed, and would have failed if that one engineer hadn’t been available or hadn’t known exactly what to do. Her skill, expertise, and dedication are worth appreciating. But her unavailability would have meant a much worse outcome, and that’s worth examining too. Too many companies celebrate the save and stop there.
The incentive nobody designed
When a company celebrates a heroic save without examining why the heroics were necessary, it sends a message. The message isn’t intentional, but it’s clear nonetheless: what gets valued is the dramatic rescue, not the boring preparedness work that would have made the rescue unnecessary.
Over time, that message shapes behavior. The engineer who writes thorough runbook documentation, trains new team members on the auth system, and invests in monitoring improvements doesn’t get the same recognition as the one who swoops in at 3 a.m. and saves the day. Preparedness work is largely invisible in performance reviews. Heroic saves are memorable.
The result is a perverse incentive loop. Heroics get rewarded, preparedness doesn’t, and the company remains dependent on heroic saves because nobody is investing in the alternative. This isn’t because anyone explicitly decided that preparedness doesn’t matter. It’s because the reward system is quietly rewarding the wrong thing, and nobody has noticed because the heroes keep delivering results. Until they don’t.
In my experience, this is one of the most common patterns in companies that are struggling with incident management. They have talented, dedicated people who keep delivering heroic results, and because the results keep coming, nobody realizes there’s a growing structural problem underneath.
The hero as single point of failure
The incentive loop creates a second problem. The hero gradually becomes a bottleneck and a single point of failure. When that engineer is on vacation and the next auth system incident hits, the team might spend hours just figuring out what’s going wrong, let alone fixing it. When they eventually leave the company (as they likely will; heroes tend to burn out), the team discovers that critical knowledge walked out the door with them.
I see this pattern regularly in my consulting work. In a company’s most serious incidents, it keeps turning to the same handful of heroic engineers. Those engineers are talented and committed, and their involvement has genuinely saved the company from significant damage. Everyone involved with incidents knows who they are, and breathes a sigh of relief when they join an incident channel. But the company has never seriously examined what its response capability looks like without them. The term that often comes up to describe these people is “indispensable,” which is really another way of saying that the company’s incident response capability depends on specific individuals’ availability.
Why the problem stays hidden
The most insidious aspect of this pattern is that it’s invisible to leadership for as long as the heroes keep delivering. Companies at the earliest stages of incident management maturity often don’t realize they’re at risk. Leadership sees consistently good outcomes and assumes the company has strong incident response, when what they actually have is strong individuals (and a certain amount of good luck).
By the time the fragility surfaces, the gap between where the company thought it was and where it actually was can be startling.
Heroic is a growth stage, not a compliment
When I assess incident management capabilities for my consulting clients, one of the dimensions I evaluate is program maturity: where is this company on the growth path from ad hoc response to reliable organizational capability? The first stage on that path is called “Heroic.” It isn’t meant to be flattering. It means that incident response quality is a property of specific talented individuals rather than a property of the company. When those individuals are available, things go well. When they’re not, things go sideways.
Every company starts here. The question is whether they invest in growing past it, converting individual capability into organizational capability. That transition is what the rest of the maturity model describes, and it’s the core of what effective incident management programs are designed to do.
What to recognize instead
None of this means companies should stop recognizing heroic contributions when they happen. When someone saves the day at 3 a.m., thank them. But also investigate why the heroics were necessary, and invest in the answers. That’s a form of recognition too: it says the save mattered enough to learn from.
To move from “Heroic” to higher levels of organizational capability, you need to shift what gets sustained recognition. Recognize the work that makes heroic saves unnecessary: the runbooks, the training, the well-coordinated responses where nobody had to be heroic.
If an engineer spent much of their quarter writing runbooks, training new responders, and coordinating incident responses, recognize that work: in performance reviews, in public acknowledgment from leadership, in awards and bonuses. If you don’t, you’re telling your organization that the only incident management work worth noticing is the dramatic save.
The goal is to make effective incident response something the company can do reliably, regardless of who happens to be on call. Heroes are still welcome, and still admired. They just shouldn’t be required.
by Brent Chapman | Jul 14, 2026 | Uncategorized
Every fire department has a training program. Big-city departments have entire training divisions; even small volunteer departments that can’t spare anyone full time still name a training officer. Not because training is the department’s mission, but because maintaining the capability to do the mission requires sustained, dedicated attention.
New recruits need to be brought up to speed. Everyone needs to learn about evolving techniques and new equipment. Procedures need to be updated as building codes and materials change. Hard-won lessons from past incidents would survive only as stories told around the kitchen table; the fire service has a strong storytelling tradition, and its legends and cautionary tales carry real value, but oral history is hard to study, standardize, and train on.
Maintaining operational capability is itself a job, distinct from the operational work it supports, and fire departments size the role to the department rather than leave it unassigned.
Many software companies haven’t learned this yet. They invest real effort in building an incident management process. They define severity levels, write runbooks, designate incident commanders (ICs), set up communication channels. The project might take weeks or months of focused work, often driven by someone who cares deeply about doing it right (and often done in their “spare time”). When it’s done, it works, at least for a while. Incidents get declared. ICs run the response. Post-incident reviews happen. Everyone takes it for granted.
Then the person driving it gets promoted, or moves to another team, or leaves the company. The process, which was never really institutionalized because it didn’t need to be while that person was carrying it, begins to decay. Not catastrophically, but more like a garden nobody is tending any more: it doesn’t collapse overnight, it just slowly fills with weeds until one day you look up and realize the original design is barely recognizable.
The training materials haven’t been updated since the initial rollout. New engineers join but never go through incident training because nobody is scheduling it anymore. The severity level definitions still describe one product, but the company now has three. The IC rotation is running on the same six people it started with, even though the engineering team has doubled in size. The post-incident review template still references a tool the company stopped using a year ago.
None of these are crises on their own. Each one is easy to defer. But they compound, and the cumulative effect is that the process on paper bears less and less resemblance to what actually happens during incidents. In my experience, six months is roughly how long institutional momentum carries before the absence of active stewardship becomes visible in the quality of your incident responses. And growth accelerates the decay: the company simply grows away from the process, and nobody’s job is to notice.
This is what happens when you have a process but not a program.
A process is not a program
A process is a set of documented procedures: how incidents get declared, who fills which roles, what communication channels to use, how to run a post-incident review. A process can be written down, trained once, and followed.
A program is the organizational structure that develops, maintains, evolves, and champions the process over time. It’s the thing that keeps the process alive.
Many companies build the process and assume they’ve built the program. They haven’t. They’ve written a document, and documents don’t train new hires, don’t recruit for on-call rotations, and don’t update themselves when the company reorganizes around them. People do those things, and it only happens reliably when it’s actually somebody’s job.
“Everybody owns it” means nobody owns it
When I ask companies who owns their incident management program, the most common answer is some version of “we all do” or “the engineering organization as a whole.” This sounds collaborative. In practice, it means nobody has the explicit responsibility, the dedicated time, or the institutional authority to keep the process alive.
This organizational challenge isn’t unique to incident management. Companies that are serious about security don’t say “everybody owns security” and leave it at that. They assign ownership because shared responsibility without explicit ownership means the work doesn’t get done.
Incident management is the same kind of organizational capability. It needs someone whose actual job, not just their passionate side interest, is keeping it healthy.
What a program actually does
When I talk about an incident management program, I mean ownership of the full lifecycle of the capability, not just the procedures themselves. That includes keeping everything current as the company grows and changes: process documentation, severity definitions, escalation paths, tooling, runbooks.
It includes running a training pipeline so new hires are prepared before their first real incident, not thrown into the deep end during it. It includes maintaining the incident commander corps: recruiting new incident commanders, nurturing their development, supporting healthy on-call rotations across teams, and recognizing the people who do this demanding work. My former Slack colleague Scott Nelson Windels likens this to the farm teams and academies that elite sports clubs run: the point isn’t just fielding today’s roster, it’s making sure capable players are always coming up to fill it next quarter, too.
And it includes owning the post-incident review process and looking across incidents for patterns that no individual team would spot on their own. It includes tracking whether the process is actually being followed, and investigating when it isn’t, not to punish people, but to understand whether the process needs to change.
No single component is enough on its own, and no component stays healthy without sustained attention.
The good news
Building a program doesn’t require hiring a large team or creating a new department. At many companies, especially smaller ones, it starts with one person who has explicit ownership and dedicated time. What matters is that the responsibility is named, visible, and institutionally supported, not just assumed.
Here’s a quick test. Ask who owns your incident management program. Not who wrote the process, and not who ran the last big incident, but who is accountable, today, for whether the training is current, the rotations are staffed, and the severity levels still match the product. If the answer is a name, the follow-up question is what happens when that person leaves. If the answer is “everybody,” or someone who left the company last year, the process is quietly withering. And if you have a program but it would collapse without you, you haven’t finished building it yet.
The fire department didn’t name a training officer because it had extra budget. It named a training officer because it understood that maintaining a capability requires ongoing investment. The alternative, assuming trained firefighters stay trained and procedures stay current without anyone specifically owning those things, is how capabilities quietly erode until they fail when you need them most.
I’m writing a book on Incident Management for DevOps and SRE. If you’d like to know when it’s available, and get occasional updates along the way, you can sign up at im4ds.com.
If your company needs help building its incident management program, that’s the focus of my consulting practice at Great Circle.
by Brent Chapman | Jul 7, 2026 | Uncategorized
In the middle of a major incident, a senior leader joins the response channel and posts something like this:
“Let’s try to get this resolved in the next 10 minutes, please!”
They mean it as encouragement. Maybe they’re feeling pressure from their own leadership, or from a major customer, or both. Maybe they know the CEO is worried about a contract renewal call in an hour with a customer already frustrated with reliability. They want the team to know this incident matters. Posting a rallying message feels like leadership: it’s visible, it’s supportive in intent, and in normal day-to-day work, rallying the team really is a valuable leadership skill.
During an incident, it usually backfires.
What the responders actually hear
In nearly every incident I join, the responders are already working as fast as they feel they safely can. They don’t need to be told the incident matters; they’re the ones who got paged, who are staring at dashboards, who are juggling three competing hypotheses about what’s going on. When a leader urges that team to go faster, the message they receive isn’t “we believe in you.” It’s “you’re not working hard enough.”
And if the team is already at their limit, a call for more speed can’t add speed. It can only add anxiety. Anxiety during an incident is expensive: responders start splitting their attention between the problem in front of them and the audience watching them, and split attention is something that a complex technical investigation can’t afford. The pep talk was meant to help the team focus. It does the opposite.
The trouble with “10 minutes”
The arbitrary deadline makes it worse. Why 10 minutes? Where did that number come from? The responders don’t know. Often the leader who posted it doesn’t know either; it just sounded suitably urgent.
But now the number is sitting in the channel, and every responder is doing math against it. If the team resolves the incident in 12 minutes instead of 10, did they fail? Nobody can answer that, which means the deadline has created a test that everyone can feel and nobody can pass. Some part of each responder’s attention now goes to the clock, and to the question of how this will look to a senior leader who’ll have a big say in their next performance review. None of that attention is going to the outage anymore. The team has been handed a no-win condition in the middle of an emergency, by someone who was trying to help.
How experienced commanders convey urgency
I’ve spent a lot of time around public safety incident command, and one of the things that struck me early is what you don’t hear on the radio at a working fire: nobody broadcasts “let’s try to knock this fire down in the next 10 minutes, please!” to the crews working the fire.
What you do hear is information. “We have a report of a person trapped on the second floor” changes how the crews operate, instantly, without anyone being exhorted to care more. On the fireground, commanders convey urgency through facts and objectives, because facts and objectives change what responders do. Cheerleading doesn’t.
Even with the arbitrary number removed, “let’s wrap this up quickly, team!” is not actionable, because it doesn’t tell anyone what to do differently. It changes the mood, but probably not for the better, without changing a single decision.
Urgency is information, not exhortation
The leader’s urgency is usually genuine, and often there’s real business context behind it. That context is valuable. But it needs to be delivered as information, to the right person, through the right channel.
The right person is the incident commander (IC), the person coordinating the response. The right channel is a private one. “The CEO has a contract renewal call with our largest customer in an hour; they’re already frustrated with our reliability, and this outage isn’t going to help” is genuinely useful: the IC can prioritize mitigations that affect that customer’s services, loop in the account team, or prepare a status update the CEO can reference on the call. “Legal needs to know by end of day whether customer data was affected, so they can meet the notification deadlines in our customer contracts” is useful in the same way. The IC can act on information like that.
What to do instead
For senior leaders: when you feel that urge to rally the troops, pause and apply a simple test: can the responders use what you’re about to post to make any decision better? If so, you have real business context, and you already know where it goes: to the IC, privately. If not, it might indeed change the mood, but probably not for the better. And if the honest answer to what’s driving your urgency is “I’m anxious and I want them to know I’m paying attention,” then the most supportive thing you can do is trust the team and stay out of the channel. The most valuable contributions senior leaders make during major incidents mostly happen outside the response channel anyway: clearing roadblocks and handling the stakeholders who would otherwise be pestering the responders for updates.
For incident commanders: when one of these pep talks lands in your channel, don’t respond defensively, but don’t ignore it either. Acknowledge it briefly, then follow up with the leader privately: “Is there specific business context driving that timeline? If so, it would help me to know what it is.” Most of the time there is something behind it, and that question converts an anxiety-inducing exhortation into information you can actually use. You’ll also be quietly teaching your leadership how to engage with the next incident.
For everyone else on the response: you don’t need to respond to the pep talk, and it doesn’t change your priorities unless the IC says it does. During an incident, you take your cues from the IC, not from voices outside the response, no matter how senior. Messages like this are the IC’s to handle, and now you know how they’ll handle it.
The urgency itself was never the problem. Every incident deserves urgency. The problem is urgency delivered as pressure instead of as information, because pressure makes responders slower and more mistake-prone just when the company most needs them sharp.
I’m writing a book about all of this: Incident Management for DevOps and SRE. If you’d like to hear when it’s available, you can sign up at im4ds.com.
Need help preventing, preparing for, responding to, and learning from incidents? That’s the focus of my consulting practice at Great Circle.
Recent Comments