by Brent Chapman | Mar 31, 2026 | Uncategorized
It’s mid-morning in Europe, and your customers are complaining about stale data. It’s 2 AM for your on-call engineers. Your European customer care team noticed the surge in complaints and paged the on-call incident commander (IC). The IC pulled up the dashboards, saw a spike in processing errors on the data ingest pipeline, and paged the data engineering, platform, and networking on-call engineers. Now the IC is coordinating with three responders, all trying to figure out what’s gone wrong, and whether the problem is actually even worse than it appears. And on top of all that, the IC is expected to write a customer-facing status page update.
The result is predictable. The update either doesn’t happen at all, or it reads something like: “Elevated error rates on the ingest pipeline due to Kafka consumer lag.” Which is perfectly accurate, and perfectly useless to 99% of the people reading it.
This is one of the most common patterns I see in my consulting work, and it’s worth understanding why it happens, because the fix isn’t “train your engineers to write better status updates.” The fix is to stop asking your engineers to write them, and start asking the right people instead.
The translation problem
Here’s the core issue: incidents generate information in a language that most of your company doesn’t speak, and most of your customers don’t either.
When responders are working an incident, they talk about services, endpoints, error rates, deployment pipelines, and database replicas. They refer to systems by internal codenames. They describe impact in terms of error budgets and percentile latencies. This is exactly the right level of detail for the people doing the troubleshooting, and exactly the wrong level of detail for almost everyone else.
Your customer support team needs to know which customer-visible features are affected, described in the language customers use. Your sales team needs to know whether the demo environment is impacted and what to say to the prospect they’re meeting in an hour. Your executives need to know the business impact: revenue at risk, customers affected, estimated time to resolution. Your customers need to know that you’re aware of the problem, you’re working on it, and roughly when it will be fixed.
None of these audiences need to know that Kafka consumer lag on the ingest pipeline is causing records to back up in the processing queue. They need to know that some recently submitted data may not be appearing yet, and that your team is on it.
Most companies treat this as a writing-quality problem, but it’s a translation problem. Each of these audiences speaks a different dialect: customer-speak, business-impact-speak, relationship-speak, risk-speak. Your engineers are fluent in engineering-speak, which is precisely what you want from them during an incident. Expecting them to also be fluent in four other dialects, under pressure, at 2 AM, is setting everyone up for disappointment.
Consider how much translation is involved in turning “Kafka consumer lag on the ingest pipeline. Stuck consumer group identified; restarting. Backlog clearing, ETA 30min” into “Some users may notice that recently submitted data isn’t appearing yet. Our team has identified the cause and is deploying a fix. We expect this to be resolved within the next 30 minutes.” Internal service names became customer-visible symptoms. A technical explanation became “identified the cause.” An engineering action became an estimated timeline. That’s the work, and it’s work that customer-facing people do better than engineers.
The right people for the job
The fix is structural. Staff your incident communication functions with people who already speak the audience’s language, and teach them enough incident-speak to do the translation.
A customer support lead who has spent years talking to customers can translate “Kafka consumer lag on the ingest pipeline” into “some recently submitted data may not be appearing yet” far more effectively than an engineer who has never staffed a support queue. An executive liaison who understands how the C-suite thinks about risk can distill a technical sitrep into the three sentences the CEO actually needs, without the incident commander having to figure out what those three sentences are.
Public safety professionals figured this out decades ago. In the Incident Command System used by fire departments and other emergency services, the Public Information Officer (PIO) is a defined incident role with specific training. The principle is straightforward: the people fighting the fire are busy with that, and you want someone else, with special training and experience, speaking to the reporters and the public. The skills are different, the priorities are different, and mixed messaging is dangerous.
What this looks like in practice
The software world’s equivalent to the PIO is an incident role of Communications Lead, or a set of liaison roles, each serving a specific audience.
The simplest version: when a significant incident is declared, someone from your customer-facing team joins the response as a communications liaison. They follow the technical discussion, work from the incident commander’s sitreps, and translate updates into customer-appropriate language for the status page and support team. The IC reviews external communications for accuracy, but the drafting and posting are done by someone who knows how to talk to customers.
This has a second benefit that’s just as important: it frees the incident commander to focus on the response itself. The IC’s attention is one of the scarcest resources during an incident. Every minute they spend crafting a status page update is a minute they’re not spending on coordination, decision-making, or thinking about what they might be missing. Delegating communication beyond the response to someone better suited for the work is better for the IC, better for the response, and better for the customers.
The same principle extends to every other audience. A support update channel where a customer care liaison posts translated, agent-ready talking points. A senior management update path where an executive liaison posts business-impact summaries. A sales notification that flags which key accounts are affected, so reps know before their next customer call. A broad notification to legal, finance, and HR when a significant incident is declared, so each function can self-select whether to engage based on their own expertise, rather than waiting for the IC to figure out whether this incident has regulatory implications.
Three things customers actually want to know
Once you have the right people doing the translating, what they actually need to communicate is simpler than you might expect.
In my experience, what customers truly want to know during an incident comes down to three things: that you’re aware of the problem, that you’re working on it, and, if possible, when it will be fixed. If you cover those three, most customers will be satisfied to let you work the problem in peace.
That’s it. They don’t need the technical details. They don’t need the internal coordination. They don’t need the investigation narrative. All of that is for your team, not for them.
The worst failure isn’t saying the wrong thing
The most damaging communication failure during an incident isn’t saying the wrong thing, it’s saying nothing at all.
A status page that reads “All Systems Operational” while your customers are seeing errors. A support team that has no information to share. Executives finding out about the outage from social media. Each of these erodes trust in a way that even the most jargon-filled status update doesn’t, because silence communicates something very clearly: either you don’t know there’s a problem, or you know and don’t care enough to say anything.
Even an imperfect update is better than no update. Share what you know, be honest about what you don’t, and commit to a cadence for further updates. But calibrate the cadence to the pace of the incident; repetitive content-free “we’re still working on it” updates become noise rather than reassurance. (This is another reason to have skilled communicators handling the job. When I was leading incident management at Slack, the Customer Experience team seemed to have at least a dozen ways of saying “still working on it” without repeating themselves.)
Customers and stakeholders can forgive a rough update. They have a much harder time forgiving radio silence.
This isn’t a problem Engineering can solve alone
Here’s where this gets real: incidents don’t wait for business hours. If your company has 24/7 on-call coverage for engineering, but your customer support and communications teams work 9-to-5, then at 2 AM on a Saturday your engineer is right back to writing status page updates, because there’s nobody else available to do it.
To solve this, the conversation needs to shift from Engineering’s process to the company’s commitment. Staffing incident communication properly means that Support, Customer Success, or whoever owns customer-facing communication needs their own 24/7 on-call rotation, or at least an escalation path that works at 2 AM, even if that means the Director of Customer Care is effectively always on call for high-severity incidents.
It also means those teams need to be included in incident response training (though not necessarily the full responder training that engineering gets; a focused one-hour session on their role during incidents may be all they need). And it means there needs to be a staffing conversation, and probably a budget conversation, with a VP outside of Engineering who hasn’t thought of incident response as their problem before.
That’s uncomfortable, but it’s also clarifying. If your company truly believes that customer communication during incidents matters, then the teams who are best at customer communication need to be available when incidents happen. If those teams aren’t willing to staff for it, that tells you something about how seriously the company takes it, which is useful information in itself.
For the engineering leader reading this: you shouldn’t try to fix this alone (and you probably can’t anyway). What you can do is make the case. The argument is straightforward: Engineering has invested in 24/7 incident response capability because incidents don’t respect business hours. Customer communication during those incidents is at least as important as the technical response, and it requires a different skill set. The same logic that justifies an engineering on-call rotation justifies a communication on-call rotation.
If the company isn’t willing to make that investment, then it’s making a conscious choice to accept poor customer communication during off-hours incidents, and everyone should be honest about that tradeoff.
The real question is organizational design
The question to ask isn’t “how do we train our engineers to write better status page updates?” The question is “who in our company already has the skills to communicate with each of these audiences, and how do we bring them into our incident processes?”
That’s an organizational design question, not a training question. And it’s the kind of question that separates companies that handle incidents well from companies that just handle the technical parts well.
Remember that European customer care team from the opening? They were the ones who noticed the problem. They were the ones fielding the complaints. They already knew how to talk to the affected customers. All they needed was a seat at the table, so they could translate what the engineering team was seeing. That’s not a big ask. But it’s one that most companies have never thought to make.
I’m writing a book about incident management for software organizations, drawing on my experience building and leading incident management programs at companies like Google, Slack, and others. If you’d like to hear about it when it’s available, visit im4ds.com.
If your organization is wrestling with incident communication or other incident management challenges, I do consulting and training on these topics. You can find out more at greatcircle.com.
by Brent Chapman | Mar 24, 2026 | Uncategorized
Spontaneous swarming of responders might seem like a nuisance that breaks our tidy mental models of incident response, but it’s actually very powerful. It’s something to facilitate and encourage, not simply tolerate.
Most of us expect incident response to be fairly linear: an on-call engineer gets paged, decides an incident is called for, and pages an incident commander. The IC spins up an incident channel and pages a couple of additional responders. So far, so good.
What surprises many people is that, within minutes, a bunch of additional reinforcements often begin swarming the incident, without having been paged. A leader at one company I worked with described this as their “white blood cell response”: something goes wrong, and people spontaneously converge to help however they can.
Why do these volunteers show up? For different reasons. Some are engineers who recognize the symptoms from something they’ve seen before, or who know that their systems might be involved. Some are from customer care, account management, or product, keeping an eye on the incident to see if it’s going to affect their areas of concern.
Many of them aren’t there to actively investigate; they’re monitoring, ready to speak up if they have something to contribute. This is one of the advantages of running incidents in Slack rather than on a Zoom call: people can follow along and speak up without disrupting the response. They don’t have to wait for a pause in conversation or work up the courage to interrupt, which in my experience causes many useful observations to go unsaid.
What these spontaneous responders all have in common is that they’re busy experts who have decided that the incident is worth their attention. That’s an organizational strength worth recognizing, developing, and leveraging.
So how do you facilitate and encourage swarming? Start by understanding how incidents actually unfold.
The model vs. the reality
Most incident management processes are built around a tidy model: something happens, an incident is declared, the incident commander pages the right people, those people respond, and the organized response proceeds from there. It’s a useful model, and it describes the formal structure of incident response well enough, but like all models, it has its limits.
Real incidents are messier than that. People find out about problems in all kinds of ways. Engineers start investigating before an incident is even declared. Volunteers show up before the IC has finished their first round of pages. By the time the formal response is organized, a significant chunk of the actual response is already underway, whether it’s coordinated or not.
Often the swarm forms without any incident tooling involved at all. An engineer investigating a problem asks a coworker for help. That coworker pulls in another. Before long, there’s an ad hoc team working on an urgent, significant problem in whatever random place the discussion started, usually some team’s day-to-day channel, where it’s intermingled with “where are we going to lunch today?” and bot posts about new code PRs and deploys. Eventually someone asks “hey, maybe we should be treating this as an incident?” (Pro tip: the answer to that question is almost always an emphatic “yes!”)
When I was leading incident management at Slack, we maintained a standing channel called #incident-next, ready for whoever needed it. Anyone investigating a concern could direct people there to start collaborating in a focused space, separate from whatever team channel the discussion started in. When an incident was declared, the channel got renamed to become the incident channel, preserving all the context, the investigation history, and the people who were already there, and a new empty #incident-next was created to be ready for the next one. This is a simple mechanism that any company could implement today, and it directly supports swarming: people converge, start investigating, and the formal incident declaration catches up to what’s already happening. The response is already underway when the structure arrives to support and harness it.
Most companies design their process only for the tidy model and then get frustrated when reality doesn’t match. A better approach is to design for how incidents actually unfold, which means designing with swarming in mind as a possibility, or even something to be encouraged.
Why swarming works
In a purely command-driven response model, someone has to figure out which teams are relevant, page the right on-call engineers, wait for them to respond, brief them on the situation, and assign them work. Whether that’s the incident commander or a tech lead, every step is bottlenecked through one person’s knowledge and judgment. If they page the networking team but the problem turns out to be in the caching layer, time is lost while they regroup and page a different team. They can only route expertise as well as they can diagnose the problem, and in the early minutes of an incident, the diagnosis is often wrong.
Swarming bypasses that bottleneck. The people with relevant knowledge self-select in, often before the IC or TL even knows their expertise is needed. The engineer who made a related change earlier that day, the one who saw a similar failure pattern last month, the one who happens to know that the caching layer was acting strangely yesterday, they show up because they can see that something is wrong and they recognize that they have something to contribute. Nobody leading the response could have paged all the right people that quickly, because nobody could have known who the right people were.
On a small incident, this is often all you need. A handful of knowledgeable people converge, naturally divide the work, and the problem gets solved quickly, sometimes faster than a formal response could even get organized.
On larger incidents, swarming gets the right people into the room faster than any paging process could. The IC still coordinates the response, but instead of spending the first forty minutes figuring out who to call and waiting for them to respond, they’re organizing people who are already there and ready to work. That’s a fundamentally better starting position. The trick is to make sure that the room is ready for the swarm.
When it isn’t, the same instinct that drives swarming produces freelancing instead: people investigating on their own, without coordinating, because there was no organized response to join. Same motivated people, very different outcome.
Facilitating the swarm
If swarming is this effective, why doesn’t it work well at every company? First, people have to feel safe showing up uninvited. In a culture where volunteering in an incident means risking blame if things go sideways, the swarming instinct gets suppressed before it starts. That’s a deeper problem than process design can solve, but it’s worth naming.
Assuming the culture supports it, two practical things determine whether swarming works:
Can people find the response? When someone hears about a problem, whether it’s a teammate mentioning that checkout is broken, an alert in their team’s channel, an error spike on a dashboard, do they also hear that there’s an organized response underway?
Maybe this is an #all-incidents channel where every declared incident is automatically announced with a link to the incident channel. Active incidents surfaced on the dashboards people are already checking. A predictable naming convention for incident channels, so that anyone can type #inc- in Slack and use auto-complete to see what’s active right now. Integration with your chat platform so that people who are talking about the problem in their team channels get a pointer to the organized response.
The goal is to make sure that everyone who hears about the problem, however they hear about it, also learns where people are converging.
Then, can they easily contribute? Once people find the channel where the response is happening, they need a clear on-ramp. A pinned message with the current situation, who’s filling what role, and what’s being investigated. A clear expectation that new arrivals announce themselves and what expertise they bring, then self-brief by reading the channel and the latest situation report before asking for an assignment. Enough structure that someone can get oriented in a few minutes and start contributing, rather than facing a wall of scrollback with no obvious way to plug in.
None of this is complicated. But it does require deliberate design. The swarming instinct happens naturally. The infrastructure that makes swarming productive doesn’t.
Invest in the instinct
If your team is already swarming incidents, you’ve got something invaluable: people who care enough to show up. The rest is infrastructure: making the response visible, making it easy to join, and giving the IC the tools to organize a growing swarm on the fly.
You can’t manufacture that instinct. But you can make it dramatically more effective.
Swarming is one of the topics I cover in depth in my forthcoming book, Incident Management for DevOps and SRE. If you’d like to know when it’s available, sign up at im4ds.com. And if your company needs help with incident management right now, my consulting practice is greatcircle.com/im.
by Brent Chapman | Mar 17, 2026 | Uncategorized
Most companies have some notion of what an “incident” is: a significant problem that requires an urgent response and involves multiple responders. But having a definition and actually using it are two different things. Teams at most companies I’ve worked with know perfectly well what an incident is, at least in theory. In practice, they still hesitate to declare one, for reasons that have nothing to do with the definition.
The pattern is remarkably consistent. An engineer sees something that looks wrong and thinks “maybe it’s not that bad.” A team notices a problem that could be an incident, but nobody wants to “bother” the on-call incident commander over what might turn out to be nothing. Someone suspects they should declare, but waits for a more senior person to make the call, assuming it’s not their place. A customer care agent gets their third chat this hour about the same issue, and wonders if it’s just a coincidence. An account exec gets a curious “are you folks having any problems today?” from their contact at a major customer, but isn’t sure whether or how to escalate. Across the company, people are smelling a whiff of smoke, but nobody is pulling the fire alarm.
When I assessed one company’s incident management program, stakeholders estimated they were capturing only 75-80% of actual incidents. The rest were being handled informally, in ad hoc threads and side conversations, without the coordination, communication, and documentation that the incident process provides. That’s typical, in my experience; if anything, 75-80% is better than most.
The interesting question isn’t whether your team waits too long, or avoids declaring at all. They almost certainly do. The interesting question is why.
It’s Not a Judgment Problem
The obvious explanation is that people don’t know where the line is. If only we had clearer criteria, the thinking goes, people would declare at the right time. So companies invest in decision trees and flowcharts and severity matrices, but the problem doesn’t get better.
It doesn’t get better because unclear criteria aren’t the real issue. The real issue is that your company has made declaring an incident costly and risky for the person who does it.
Think about what happens when someone declares an incident at your company. I’ll bet it looks something like this:
They’re committing real time and attention, and not just for themselves. They’ve just pulled themselves and several colleagues away from whatever they’d planned to be working on, and added several hours of incident-related work to each of their already-overflowing plates.
There’s no lightweight way to raise an urgent concern. There’s no easy way to say “I think something is wrong right now” and get someone experienced to look into it quickly. Filing a bug or ticket puts the problem in a queue; the only way to get an immediate response is to formally declare an incident, with all the overhead that entails. The person who notices the problem has to decide that it’s an incident, assess how bad it is, and figure out who to page. That’s a lot to expect from whoever randomly happens to notice first, and almost guarantees that nothing happens until somebody senior enough or confident enough eventually notices the problem.
They’re triggering a heavyweight process. Declaring the incident guarantees that a heavyweight post-incident review process kicks in, with mandatory documentation and required meetings. The incident gets counted, and someone in leadership is tracking that count, wanting it to trend downward.
They’re paying a career cost. The time engineers spend responding doesn’t earn them any credit on their performance review; at best it’s invisible, and at worst it’s counted against the “real” work they didn’t finish. Their sprint commitments don’t get adjusted. Their manager doesn’t say “I see you spent 15% of this quarter helping with incidents, so let’s recalibrate your goals.” The incident work just disappears into an unacknowledged gap between what they delivered and what was expected.
Engineers aren’t oblivious to these incentives, even if they couldn’t name them. They’re responding to them rationally. When declaring an incident is costly, for themselves and for the coworkers they’d be pulling in, people unconsciously raise their internal threshold for what’s worth declaring. They wait a little longer, hoping the problem resolves itself. They try to fix it quietly before anyone notices. They let someone else make the call.
Declaring Early Beats Declaring Late
Here’s what companies often overlook: the costs of declaring too early and declaring too late are not the same.
If someone declares an incident and it turns out to be a false alarm, the cost is small. A few people spend a few minutes getting oriented, realize the situation is under control, and stand down. In fact, it’s not really a cost at all; your team just got a bit of practice with the incident process, which is valuable in its own right.
If someone doesn’t declare an incident and the problem turns out to be serious, the costs are large. Customers are affected longer. The blast radius expands. What could have been a contained, quickly resolved incident becomes a prolonged outage. And the longer a problem goes unaddressed, the harder it gets to recover from.
One of my flight instructors taught me a lesson that I apply constantly in incident management: “If you wonder whether you’re running out of fuel, you’re running out of fuel.” If your subconscious is even raising the question, it’s telling you something. The same principle applies here. If someone is wondering “should we be treating this as an incident?”, the answer is almost certainly “yes!”
Fix the Incentives
If your team is slow to declare, the fix usually isn’t better decision flowcharts. It’s examining the incentives, both explicit and implicit, that your company has built around incident declaration, and changing the ones that inadvertently punish people for doing the right thing. Here’s where to start.
Make declaring cheap. Not every incident needs a full post-incident review. Lightweight incidents that resolve quickly should have a lightweight process. If every declaration triggers the same heavyweight machinery regardless, people will avoid declaring in order to avoid the machinery.
Separate reporting from declaring. When someone sees something that might be an incident, they should be able to flag it quickly, without having to determine the severity, identify which teams to page, or commit to a formal declaration.
Think of it like calling your local emergency number (911, 999, 112, or whatever your country’s is): the caller describes what they’re seeing, and the person who answers (who has had special training and lots of experience in evaluating reports of possible emergencies) decides what response is called for. The person who reports the problem doesn’t bear the weight of all those “is it an incident? how severe? who do we page?” decisions. They just raise the alarm, and an expert (typically an on-call incident commander) takes it from there.
Stop fixating on incident counts. When leadership treats the number of incidents as a metric to drive down, the predictable result is that people stop declaring. The number of incidents you declare should not be a target. In fact, a rising count can be a healthy sign: it may mean people are becoming more comfortable with the process and using it more. Signs of success such as more customers, more features, more employees, and more usage can all lead to higher incident counts.
Recognize incident work as real work. Time spent responding to incidents needs to be visible in performance ratings, bonuses, and promotion decisions. Not as a footnote, not as an “also did,” but as a genuine contribution to the company. If the only work that counts is feature delivery, and working on incidents is a distraction from that, then your best engineers will subconsciously but rationally avoid incident work.
Watch for mixed messages. Companies often undermine their own stated values without realizing it. A leader pressures a team about incident counts while simultaneously asking why problems aren’t caught earlier. A manager expresses frustration about “unnecessary” declarations while wondering why the team doesn’t escalate fast enough. A VP asks why incidents aren’t caught sooner, then questions why their team is spending so much time on incidents. The contradictions send a clear message about what’s actually valued, regardless of what’s written in the incident management policy.
These mixed messages aren’t limited to Engineering. Noticing and resolving these contradictions, at every level and throughout the entire company, is what makes timely declaration a cultural reality instead of just an aspiration.
When in Doubt, Declare
The next time someone is wondering “is this bad enough to declare?”, you want “yes!” to be an easy call. Not because you’ve written better criteria, but because you’ve built a system where the cost of declaring is low, the process is lightweight, and the culture rewards raising the alarm.
All your incentives and processes should align to reinforce one simple principle, without fear or reservation:
When in doubt, declare!
This is one of the foundational questions I tackle in my book, Incident Management for DevOps and SRE, which I’m currently writing. If you’d like to be notified when it’s available, sign up at im4ds.com.
If your company is wrestling with these questions right now and doesn’t want to wait for the book, my consulting practice can help.
by Brent Chapman | Mar 10, 2026 | Uncategorized
Ask six people at the same company what counts as an incident, and you’ll likely get six different answers.
The IT ops manager thinks of every ticket in ServiceNow. The on-call SRE thinks of the alert that paged at 2 AM. The head of sales thinks of the call from an angry enterprise customer. The finance team thinks of the disruptions they have to issue SLA credits for. The engineering manager thinks of last week’s outage, the one she was up until 3 AM for, acting as the incident commander and coordinating the response across three teams. The newest engineer isn’t sure, but knows they don’t want to be the one who declares one.
This confusion isn’t academic. When people in the same organization mean different things by “incident,” practical problems follow. The CFO asks “how many incidents did we have last quarter?” to forecast SLA payouts for the board report, and gets a completely different number depending on who answers. Engineering can’t tell whether the trend line is improving because the data mixes SRE pages, ITIL tickets, and multi-team coordinated responses into a single count. Someone declares an incident and half the organization thinks it’s a crisis while the other half thinks it’s Tuesday. And every attempt to improve incident management stalls because the people in the room haven’t realized they’re talking about different things.
Where the Confusion Comes From
The word “incident” has been overloaded with several overlapping meanings in the technology industry; people think they’re all talking about the same thing, but they aren’t, and the confusion causes more problems than most people realize.
ITIL has taught generations of IT professionals that an incident is any unplanned interruption to a service. Under that definition, a single user unable to log in is an incident. A slow database query is an incident. A printer jam is an incident. ITIL-trained teams sometimes process thousands of “incidents” per month through their ticketing systems. When someone from that background hears “we need better incident management,” they’re thinking about ticket queues and resolution targets, not about coordinated emergency response.
PagerDuty, the most widely-used on-call tool in the industry, has long compounded the problem through its product terminology. In PagerDuty’s data model, which stretches back over 15 years to the early days of the company, every alert that pages someone creates an “incident.” You literally cannot page a colleague without creating a PagerDuty “incident.” This means an on-call engineer who gets paged three times on a quiet Tuesday has, in PagerDuty’s language, experienced three incidents. Over time, this trains people to think of “incident” as synonymous with “page” or “alert.” PagerDuty has recognized the gap as they’ve expanded into incident response orchestration, particularly after acquiring Jeli in 2023, adding the concept of “major incidents” (a term ITIL also uses, for the same reason) to distinguish coordinated-response situations from everyday pages. But the underlying product model remains, and generations of engineers have already internalized the equation: incident = someone got paged.
Then there’s the other extreme. Some organizations reserve “incident” exclusively for the worst events they can imagine: full-site outages, data breaches, events that make the news. This sounds disciplined, but it creates a high psychological barrier to declaration. If “incident” means “catastrophe,” nobody wants to be the person who declares one for something that turns out to be just a blip. So people hesitate, they wait for clearer signals before acting, and by the time someone finally says the word “incident,” the situation has been burning unchecked for longer than it needed to.
Those are all legitimate uses of the word, but they’re answering different questions. The ITIL definition tells you what to put in the ticketing system. PagerDuty’s tells you what to attach a page to. Finance’s tells you what catastrophes will trigger an SLA payout. None of them help you decide when to urgently pull people together and mount a coordinated response.
We’re not saying those other definitions are wrong and ours is the One True Definition. We just want folks to be aware of the potential for confusion, and to be sure they understand from context (or explicit clarification) which version of “incident” someone is referring to.
It’s like the word “security,” which means something completely different to an information security engineer, a physical security guard, and a financial analyst trading stocks and bonds. Nobody argues about which meaning is correct; they just make sure everyone in the conversation knows which one they’re talking about.
My professional community, my own work, and my forthcoming book are about managing emergencies that require urgent, multi-person, coordinated responses. In those contexts, we use “incident” to mean a situation that is significant enough to need attention right now, urgent enough that you can’t just file a ticket and walk away, and beyond what one person can handle alone. That’s one valid, context-specific definition among several. The important thing is that everyone in a conversation is clear on which one they’re using in the moment.
Watch for It
Here’s a concrete step you can take this week: ask six people at your company what they think counts as an incident. Don’t ask in the abstract; give them three or four scenarios and ask which ones they’d call an incident. Better yet, do it in a group setting where everyone can hear each other’s answers. The spread will surprise people, and that surprise is the point.
Once your team sees the mismatch, you gain the ability to catch it in the moment. “Wait, are we all talking about the same kind of incident right now?” is a surprisingly useful sentence.
This is one of the communication challenges I explore in my book, Incident Management for DevOps and SRE, which I’m currently writing. If you’d like to be notified when it’s available, sign up at im4ds.com.
If your organization is wrestling with these questions right now and doesn’t want to wait for the book, my consulting practice can help.
by Brent Chapman | Mar 3, 2026 | Uncategorized
It happens on way too many incidents. The ad hoc team of responders is deep in discussion when somebody new joins the call or the channel, and the first thing they say is: “Hey, what’s going on?”
The incident commander (IC) stops what they’re doing and recaps the timeline, the current theories, who’s working on what, and what’s been tried so far, which can easily take a few minutes. Meanwhile, the discussion and coordination stall, other responders wait (or, worse, proceed without coordination), and the IC loses their train of thought. Then fifteen minutes later someone else joins, and it happens again.
Each interruption seems small. Cumulatively, they’re one of the biggest drags on incident response that nobody talks about. And it’s not just responders. Executives, customer care reps, sales teams, and other observers who join the channel to follow along often do the same thing, without realizing how much these interruptions add up.
The problem isn’t the people
People who ask “what’s going on?” aren’t being lazy or inconsiderate. They’re doing what feels natural: they’ve been pulled into an unfamiliar situation, they don’t know what’s happening, and they turn to the person in charge for orientation. It’s a reasonable instinct.
But it’s also an expensive one. The incident commander is typically the busiest person in the response. They’re tracking multiple threads of investigation, coordinating assignments, communicating with stakeholders, and maintaining the big picture. Every time they stop to deliver a verbal briefing, all of that pauses. And because each new arrival gets a slightly different recap depending on when they ask and what the IC remembers to mention, the team can end up with inconsistent pictures of the situation.
A new arrival who doesn’t know what’s going on has three options, none of them good. They can interrupt the IC for a recap, which is disruptive. They can wait until someone has time to bring them up to speed, which delays whatever they’re there to do. Or they can start acting on incomplete information.
There’s a better alternative: self-briefing. It’s one of the simplest, highest-impact changes a team can make to their incident response, though it does require laying some groundwork.
What self-briefing looks like
Self-briefing means that when you join an incident in progress, you orient yourself rather than interrupting to ask someone else to orient you. For observers following along from the exec team or customer care, this can be completely silent: read the channel, read the latest situation report (sitrep), and you’re up to speed without anyone even knowing you arrived. For responders, a two-message pattern lets other responders know you’re there and coming up to speed, with minimal disruption.
Message one, on arrival: “Hi, I’m Jordan from the caching team. Reading back through the channel now.”
This tells the IC and the rest of the team several important things: you’re here, you have potentially relevant expertise, and you’re getting yourself up to speed rather than asking them to stop and brief you.
Then you actually do the reading. Scroll back through the incident channel. Read the most recent sitrep, if one has been posted. Check the status document if there is one. Note who’s involved and what they’re working on. This typically only takes a few minutes, depending on how long the incident has been running and how much has happened.
Message two, when you’ve briefed yourself: “Jordan from caching, caught up. I’ve read the channel and the latest sitrep. It looks like nobody is currently investigating the CDN layer. Want me to start looking there, or is there something else that would be more useful?”
Now your first real interaction with the IC is useful rather than disruptive. You’ve demonstrated that you understand the current situation. You’ve identified a potential gap. And you’ve offered a specific proposal for what you could work on, which is much easier for a busy IC to respond to than an open-ended “what do you need?”
Why this only works with text channels
Self-briefing depends on text channels. It works well in Slack (or a similar channel-oriented text tool), but isn’t feasible in Zoom (or any voice/video-first approach).
In Slack, the incident channel gives a new arrival something powerful: control over their own depth. You can skim the last twenty messages for a quick sense of where things stand, then slow down and dig into the thread about cache hit rates because that’s your area and the detail matters to you specifically. You can click the dashboard link someone shared and look at the graph yourself. You can see who said what, which tells you who’s working on what and who to direct your follow-up questions to. You can reread a confusing message until it clicks. You can forward a specific message to a colleague and ask “what does this mean?” or “this seems critical for your team.”
With Slack, a few minutes of reading lets you go shallow for orientation and deep where your expertise is relevant, all in the same pass.
In a Zoom call, that context evaporates the moment the words are spoken. You can’t rewind a conversation. When a new responder joins a call that’s been running for forty-five minutes, those forty-five minutes of discussion are gone. And it’s not just the words. Every graph someone screen-shared, every dashboard someone walked through, every log snippet someone pasted into the Zoom chat (which late joiners also can’t see) is gone too.
With Zoom, the only option is exactly what we’re trying to avoid: asking someone to stop and recap.
“But what about transcripts?” Even if your video conferencing tool offers AI-generated “catch me up” summaries for late joiners, those give you the shallow orientation pass and nothing else. They can’t show you the graphs and dashboards people were discussing while screen-sharing. They struggle with the names of people, systems, and services that fill every incident conversation. And they don’t let you drill into the specific thread that matters most to a person with your particular expertise. An AI summary might tell you the team is investigating a caching issue. It won’t let you read the actual exchange between the two engineers who narrowed it down, click through to the dashboard they were looking at, and decide for yourself whether they’ve checked the CDN layer yet.
Some teams start out with the best of intentions, keeping the Slack channel updated alongside the Zoom call. In practice, it works for a little while, and then the person doing the updating gets drawn into the verbal conversation and stops scribing. Critical information, discoveries, and decisions never make it into text. The Slack channel becomes a sparse, incomplete shadow of what’s actually happening on the call.
This isn’t a minor inconvenience. It’s a structural barrier to self-briefing. If your primary incident communication happens on a voice call, you’ve made it physically impossible for new arrivals to orient themselves without interrupting someone. You’ve baked the “what’s going on?” problem into your communication architecture.
Periodic situation reports posted on a regular cadence by the incident commander help bridge this gap, because a good sitrep gives a new arrival a snapshot of the current state regardless of what communication tool the team is using. But sitreps are periodic summaries. They can’t let you explore the details that matter for your specific expertise. A team that communicates primarily in text gets both: the running record you can explore at your own depth in the channel, and the periodic snapshot in the sitrep.
Make it an organizational expectation
Self-briefing sounds simple, and it is. But it won’t happen consistently unless the organization establishes it as an explicit expectation, not just a nice idea. This means a few things.
Teach the pattern. Include self-briefing in your incident response training. Teach responders the two-message template. Explain why it matters. Most people will adopt it readily once they understand the reasoning; they just need to know it’s expected.
Lay the groundwork. Self-briefing depends on having something to brief yourself from. That means using a text channel as your primary communication tool during incidents, posting regular sitreps, and maintaining a status document on longer incidents. If your team’s incident communication happens mainly on a Zoom call with a neglected Slack channel on the side, there’s nothing for a new arrival to self-brief from. The expectation and the groundwork go hand in hand: each one reinforces the other.
Reinforce it from the IC role. When a new responder joins and immediately asks “what’s going on?”, the IC can gently redirect: “Welcome! Take a few minutes to read back through the channel and check the latest sitrep, then let me know what questions you have.” A few consistent redirections establish the norm quickly.
Model it yourself. When you join an incident as a responder, follow the two-message pattern even if you could get a faster verbal briefing. Especially if you’re senior. When a staff engineer or a VP visibly self-briefs rather than expecting a personal recap, it sends a powerful signal about how things work here.
The compounding benefit
Self-briefing doesn’t just help the person who arrives prepared. It helps everyone.
The IC stays focused on managing the response instead of delivering repeated briefings. The investigation maintains momentum because nobody is hitting pause to catch up new arrivals. The existing responders don’t lose context from their own work while waiting for the IC to finish briefing someone else. Observers from customer care or the exec team can follow along and update their own stakeholders without pulling anyone away from the response. And the new responder starts contributing faster, because five minutes of reading usually builds better context than a rushed verbal summary anyway.
Over the course of a large incident where a dozen people join at different times, the difference between “everyone self-briefs” and “everyone asks the IC for a recap” can easily be an hour of the IC’s time, and several hours of cumulative disruption to the response.
The technique is simple, but the impact is significant. For observers, self-briefing is invisible: just read the channel and the latest sitrep. For responders, it takes two messages and a few minutes of reading. Nobody gets interrupted. Nothing stalls. And the response keeps its momentum.
Self-briefing is one of the practices I cover in my forthcoming book, “Incident Management for DevOps and SRE.” If you’d like to hear when it’s available, sign up at im4ds.com. And if your organization needs help building effective incident management practices right now, my consulting practice is greatcircle.com/im.
Recent Comments