Most companies have some notion of what an “incident” is: a significant problem that requires an urgent response and involves multiple responders. But having a definition and actually using it are two different things. Teams at most companies I’ve worked with know perfectly well what an incident is, at least in theory. In practice, they still hesitate to declare one, for reasons that have nothing to do with the definition.
The pattern is remarkably consistent. An engineer sees something that looks wrong and thinks “maybe it’s not that bad.” A team notices a problem that could be an incident, but nobody wants to “bother” the on-call incident commander over what might turn out to be nothing. Someone suspects they should declare, but waits for a more senior person to make the call, assuming it’s not their place. A customer care agent gets their third chat this hour about the same issue, and wonders if it’s just a coincidence. An account exec gets a curious “are you folks having any problems today?” from their contact at a major customer, but isn’t sure whether or how to escalate. Across the company, people are smelling a whiff of smoke, but nobody is pulling the fire alarm.
When I assessed one company’s incident management program, stakeholders estimated they were capturing only 75-80% of actual incidents. The rest were being handled informally, in ad hoc threads and side conversations, without the coordination, communication, and documentation that the incident process provides. That’s typical, in my experience; if anything, 75-80% is better than most.
The interesting question isn’t whether your team waits too long, or avoids declaring at all. They almost certainly do. The interesting question is why.
It’s Not a Judgment Problem
The obvious explanation is that people don’t know where the line is. If only we had clearer criteria, the thinking goes, people would declare at the right time. So companies invest in decision trees and flowcharts and severity matrices, but the problem doesn’t get better.
It doesn’t get better because unclear criteria aren’t the real issue. The real issue is that your company has made declaring an incident costly and risky for the person who does it.
Think about what happens when someone declares an incident at your company. I’ll bet it looks something like this:
They’re committing real time and attention, and not just for themselves. They’ve just pulled themselves and several colleagues away from whatever they’d planned to be working on, and added several hours of incident-related work to each of their already-overflowing plates.
There’s no lightweight way to raise an urgent concern. There’s no easy way to say “I think something is wrong right now” and get someone experienced to look into it quickly. Filing a bug or ticket puts the problem in a queue; the only way to get an immediate response is to formally declare an incident, with all the overhead that entails. The person who notices the problem has to decide that it’s an incident, assess how bad it is, and figure out who to page. That’s a lot to expect from whoever randomly happens to notice first, and almost guarantees that nothing happens until somebody senior enough or confident enough eventually notices the problem.
They’re triggering a heavyweight process. Declaring the incident guarantees that a heavyweight post-incident review process kicks in, with mandatory documentation and required meetings. The incident gets counted, and someone in leadership is tracking that count, wanting it to trend downward.
They’re paying a career cost. The time engineers spend responding doesn’t earn them any credit on their performance review; at best it’s invisible, and at worst it’s counted against the “real” work they didn’t finish. Their sprint commitments don’t get adjusted. Their manager doesn’t say “I see you spent 15% of this quarter helping with incidents, so let’s recalibrate your goals.” The incident work just disappears into an unacknowledged gap between what they delivered and what was expected.
Engineers aren’t oblivious to these incentives, even if they couldn’t name them. They’re responding to them rationally. When declaring an incident is costly, for themselves and for the coworkers they’d be pulling in, people unconsciously raise their internal threshold for what’s worth declaring. They wait a little longer, hoping the problem resolves itself. They try to fix it quietly before anyone notices. They let someone else make the call.
Declaring Early Beats Declaring Late
Here’s what companies often overlook: the costs of declaring too early and declaring too late are not the same.
If someone declares an incident and it turns out to be a false alarm, the cost is small. A few people spend a few minutes getting oriented, realize the situation is under control, and stand down. In fact, it’s not really a cost at all; your team just got a bit of practice with the incident process, which is valuable in its own right.
If someone doesn’t declare an incident and the problem turns out to be serious, the costs are large. Customers are affected longer. The blast radius expands. What could have been a contained, quickly resolved incident becomes a prolonged outage. And the longer a problem goes unaddressed, the harder it gets to recover from.
One of my flight instructors taught me a lesson that I apply constantly in incident management: “If you wonder whether you’re running out of fuel, you’re running out of fuel.” If your subconscious is even raising the question, it’s telling you something. The same principle applies here. If someone is wondering “should we be treating this as an incident?”, the answer is almost certainly “yes!”
Fix the Incentives
If your team is slow to declare, the fix usually isn’t better decision flowcharts. It’s examining the incentives, both explicit and implicit, that your company has built around incident declaration, and changing the ones that inadvertently punish people for doing the right thing. Here’s where to start.
Make declaring cheap. Not every incident needs a full post-incident review. Lightweight incidents that resolve quickly should have a lightweight process. If every declaration triggers the same heavyweight machinery regardless, people will avoid declaring in order to avoid the machinery.
Separate reporting from declaring. When someone sees something that might be an incident, they should be able to flag it quickly, without having to determine the severity, identify which teams to page, or commit to a formal declaration.
Think of it like calling your local emergency number (911, 999, 112, or whatever your country’s is): the caller describes what they’re seeing, and the person who answers (who has had special training and lots of experience in evaluating reports of possible emergencies) decides what response is called for. The person who reports the problem doesn’t bear the weight of all those “is it an incident? how severe? who do we page?” decisions. They just raise the alarm, and an expert (typically an on-call incident commander) takes it from there.
Stop fixating on incident counts. When leadership treats the number of incidents as a metric to drive down, the predictable result is that people stop declaring. The number of incidents you declare should not be a target. In fact, a rising count can be a healthy sign: it may mean people are becoming more comfortable with the process and using it more. Signs of success such as more customers, more features, more employees, and more usage can all lead to higher incident counts.
Recognize incident work as real work. Time spent responding to incidents needs to be visible in performance ratings, bonuses, and promotion decisions. Not as a footnote, not as an “also did,” but as a genuine contribution to the company. If the only work that counts is feature delivery, and working on incidents is a distraction from that, then your best engineers will subconsciously but rationally avoid incident work.
Watch for mixed messages. Companies often undermine their own stated values without realizing it. A leader pressures a team about incident counts while simultaneously asking why problems aren’t caught earlier. A manager expresses frustration about “unnecessary” declarations while wondering why the team doesn’t escalate fast enough. A VP asks why incidents aren’t caught sooner, then questions why their team is spending so much time on incidents. The contradictions send a clear message about what’s actually valued, regardless of what’s written in the incident management policy.
These mixed messages aren’t limited to Engineering. Noticing and resolving these contradictions, at every level and throughout the entire company, is what makes timely declaration a cultural reality instead of just an aspiration.
When in Doubt, Declare
The next time someone is wondering “is this bad enough to declare?”, you want “yes!” to be an easy call. Not because you’ve written better criteria, but because you’ve built a system where the cost of declaring is low, the process is lightweight, and the culture rewards raising the alarm.
All your incentives and processes should align to reinforce one simple principle, without fear or reservation:
When in doubt, declare!
This is one of the foundational questions I tackle in my book, Incident Management for DevOps and SRE, which I’m currently writing. If you’d like to be notified when it’s available, sign up at im4ds.com.
If your company is wrestling with these questions right now and doesn’t want to wait for the book, my consulting practice can help.
Recent Comments