Your team burned 40 minutes in an incident chasing a ghost in the metrics. It turned out someone unaware of the incident had picked a bad time for a routine restart.
This is sometimes called “freelancing,” or “going rogue.” It’s working on (or near) the incident without being part of the organized response, and it’s one of the most common complaints I hear from engineering leaders when they talk about incident management: “Our people go off and do their own thing instead of coordinating.”
And they’re right to be concerned. Uncoordinated work during an incident is genuinely costly. Incident “freelancers” muddy the trail. Their queries bog down log systems, making other responders’ searches slower. Their investigations generate artifacts that get mistaken for symptoms of the actual problem. They make changes that mask the issue or introduce new ones. In the scenario above, the incident responders lost 40 minutes because someone’s routine restart looked, in the dashboards, like a clue.
The instinct is to treat this as a behavior problem: tell people not to freelance, put it in the incident guidelines, remind everyone in training. And if they keep doing it, escalate.
Unfortunately, in my experience, most of what gets called “freelancing” isn’t actually freelancing.
What real freelancing looks like
Real freelancing is a deliberate choice: someone knows an incident is underway, knows there’s an organized response, but decides to work the problem independently anyway. Maybe they think they’ll be faster on their own. Maybe they don’t trust the incident commander to use them effectively. Maybe they’ve had frustrating experiences in past incidents and decided it wasn’t worth the trouble.
This is a real phenomenon, and it’s worth taking seriously. When someone with relevant expertise actively avoids the coordinated response, that’s a signal about your incident management culture: something about the experience of participating is broken enough that a skilled person would rather work alone.
Fortunately, this kind of deliberate freelancing is rare. Most companies that think they have a freelancing problem actually have something quite different.
The three gaps
Go back to the ghost in the metrics caused by the routine but uncoordinated restart. The person who did the restart wasn’t working the incident independently; they weren’t working the incident at all. But why were they doing a routine restart in the middle of a Sev-1?
Maybe they had no idea an incident had been declared. Maybe they knew something was going on but didn’t think it involved their systems. Or maybe they suspected it might be a bad time, but had no way to check. Same outcome, three very different backstories, and the fix is different for each one.
When you look at the incidents where uncoordinated work caused problems, most of them trace back to one of these three structural gaps.
The visibility gap. The person didn’t know an incident had been declared. Maybe the declaration went to a channel they aren’t in, or one they haven’t caught up on yet. Maybe the alerting didn’t reach their team. Maybe they were heads-down in focused work and missed it entirely. They weren’t choosing to work outside the response; rather, they didn’t know there was a response to join.
This is the most common gap, and it’s the one that produces the most collateral damage, because the person has no reason to think their normal work might interfere with anything. They restart a service, run a migration, deploy a config change, all routine, all uncoordinated with the incident responders, and all potentially confusing to responders trying to interpret what they’re seeing in the dashboards.
The identity gap. The person is aware something is going on but doesn’t see themselves as relevant. “That’s a payments incident; I’m on the search team.” They carry on with their normal work, not realizing that both teams depend on the same cache cluster and that the real problem is there. The incident responders don’t know to warn them, because they don’t see the shared dependency either.
This gap is subtler than the visibility gap. The information about the incident reached the person; the connection to their own work didn’t.
The mechanism gap. The person knows about the incident and suspects their work might be relevant, but there’s no clear way to check. There’s no place to ask “is now a bad time for routine changes?”, no lightweight way to raise a hand and coordinate. So they make a judgment call, usually in the direction of “it’s probably fine,” and carry on.
This is the gap that frustrates well-intentioned people the most. They would have coordinated if there had been a well-understood way to do so. But the response didn’t have one, so they did the best they could with the information they had.
The plumbing fix
Unlike “real” (i.e., intentional) freelancing, all three of these gaps are plumbing problems, not people problems. They’re about whether your incident response infrastructure makes it easy for people across the company to know an incident is happening, to understand whether their work might be affected, and to coordinate without joining the full response.
What would that look like? In the everyday world, the flashing lights at an emergency scene are a broadcast signal to everyone in the vicinity. They tell passing drivers, pedestrians, and nearby work crews: something is happening here, adjust your behavior. Nobody expects individual firefighters to personally flag down every car that might drive through the scene. The lights do that job passively, at scale, without coordination.
The flashing lights can also serve responders. At an incident scene (in the US, at least), a green flashing light marks the command post, where the incident commander can be found, and arriving responders know to go there to check in for an assignment.
Most companies don’t have the equivalent of flashing lights for their incidents. The incident declaration establishes the incident channel, and the paged responders join it, while everyone else in the company carries on unaware.
The fixes are concrete and mostly unglamorous, and include:
Broad incident visibility. When an incident is declared, the notification should reach beyond the directly-paged responders. A company-wide incidents channel, automated cross-posts to team channels for affected services, a banner in internal tools: whatever fits your company’s communication patterns. The goal is that anyone doing work that might intersect with the incident has a reasonable chance of knowing about it. For significant incidents, that notification can include a simple “hold non-urgent changes, or check with the incident channel” signal.
Dependency-aware notifications. When an incident is declared for one service, teams that own connected services often don’t realize the incident might involve them. If your company maintains a service dependency map (even a rough one), use it: automatically notify teams whose services are upstream or downstream of the affected system. “Heads up: there’s an active incident involving the payments service, which depends on your cache layer” turns “not my problem” into “maybe I should hold off on that restart.”
A lightweight coordination path. Not everyone who might be affected needs to join the incident response. But they need a way to check in: “I was about to restart the cache fleet; is that going to cause problems for you?” A cultural norm that it’s OK (even expected) to ask in the incident channel turns invisible collisions into two-minute conversations.
These aren’t expensive changes. They’re the kind of infrastructure that, once built, quietly prevents dozens of 40-minute detours a year. They work because they address the actual problem: most people who end up doing uncoordinated work during incidents aren’t choosing to go rogue. They just didn’t have the information or the path to coordinate.
Real freelancing, the deliberate kind, still deserves attention. But if you’re seeing a pattern of uncoordinated work during your incidents, start with the plumbing before you start with the lectures.
I’m writing a book about incident management for software engineering companies. If you’d like to hear about it when it’s available, sign up at im4ds.com. And if your company needs help with incident management right now, my consulting practice is GreatCircle.com/im.
Recent Comments