by Brent Chapman | Jun 30, 2026 | Uncategorized
Most of the time, simultaneous incidents are no big deal. Any company with enough engineers and enough services will have multiple incidents open at the same time; when I led incident management at Slack, we had a few thousand engineers, and half a dozen concurrent incidents was not unusual for a typical weekday. Each had its own incident commander (IC), its own responders, its own channel, and they proceeded independently without anyone needing to think about the others.
Any conflicts that emerged over priority or resources could usually be worked out among the incident commanders of the individual incidents. The most obvious approaches are straightforward: a higher-severity incident takes priority over a lower-severity one, or a customer-impacting incident takes priority over an internal-only one. When only two incidents are in contention, the two ICs can almost always negotiate a decision between themselves.
Sometimes, though, negotiation among the ICs couldn’t produce a timely solution, particularly when more than two incidents were involved.
Consider a large SaaS platform with three active incidents, all with fixes ready, all needing the same deploy pipeline. Incident A is a Sev-1: the customer-facing API is returning errors for a segment of users. Incident B is also a Sev-1: a partner-facing integration is down. Incident C is a Sev-2: a background data pipeline is falling further behind, and if it isn’t addressed in the next few hours, the backlog will cascade into customer-impacting failures worse than A or B. The severity heuristic doesn’t break the tie between A and B. The customer-impact heuristic doesn’t help either: A and B are both customer-impacting in different ways, and C will be soon. With two incidents, the two ICs can talk it through. With three, each IC is advocating for their own incident, and the tradeoffs involve factors that no individual IC has visibility into: the contractual implications of A, the partner relationship at stake in B, the rate at which C’s backlog is growing. Somebody with a cross-incident perspective needs to decide the sequencing.
When I was at Slack, this kind of contention came up about twice a year. The trigger was usually the deploy pipeline for the monolith. Normally, incidents deployed fixes alongside each other and routine code pushes without any issues. But occasionally, an incident’s fix was risky enough to need exclusive use of the pipeline: a staged rollout that could take a couple of hours, where we didn’t want to bypass the staging except in a dire emergency, and didn’t want to bundle high-risk fixes for multiple incidents into the same deploy because rolling back one fix mid-deploy could derail the other. When multiple incidents each wanted exclusive use of the pipeline at the same time, the ICs were stuck.
Three options when incidents collide
When simultaneous incidents start interfering with each other, there are three options.
Leave them separate. If the interference is minor and manageable (one incident can wait an hour for the deploy pipeline, or the resource contention resolves with a quick conversation between the ICs), there may be nothing to do beyond agreeing on a path forward. Not every collision warrants escalation.
Combine them. When investigation reveals that two “separate” incidents share a common underlying cause, consider merging them into a single response. What looked like independent problems turns out to be different symptoms of the same failure, and maintaining separate responses would mean duplicating effort and coordination. But don’t be in too big a hurry to merge: concurrent incidents are common, and correlated incidents are much less common. Merging is also hard to undo; if you realize mid-response that the incidents weren’t actually related, unmerging is about as messy as reopening a closed incident (pro tip: don’t try, open a new incident instead).
Stand up a coordination layer. When multiple active incidents are genuinely distinct (different systems, different expertise needed), but they’re competing for the same scarce resources, and the individual ICs can’t resolve the contention among themselves, somebody needs to make the cross-incident prioritization calls. Keep the incidents separate but add a coordinator above them to make priority, policy, and resource allocation decisions. In the Incident Command System, this coordination layer is called Area Command.
Area Command
To see why the Area Command pattern is needed, consider a tornado outbreak that drops multiple tornadoes across a metro area in quick succession. A school gym hosting a basketball game has partially collapsed with people trapped inside. A residential neighborhood has been leveled, with reports of injuries throughout. A gas line rupture in an industrial area has started a fire that’s threatening an adjacent apartment complex. Each of these is a separate incident with its own IC and responders, but there aren’t enough responders to go around; all three incidents need the same scarce resources.
Heavy-rescue teams are needed at the school gym to reach trapped spectators, but also in the residential neighborhood where people are buried in collapsed houses. Fire engines are needed at the gas fire to keep it from reaching the apartment complex, but the residential neighborhood also has secondary fires breaking out. All three incidents need ambulances. None of the individual incident commanders is in a position to make the tradeoffs across incidents; each one is rightly focused on their own scene. Somebody above them needs to make the hard calls: which scene gets the heavy-rescue teams first? How many of the fire engines go to the industrial fire and how many get sent to the neighborhood? That’s Area Command.
The structure is deliberately lean. The Area Commander doesn’t manage any incident directly. They allocate scarce resources across incidents, set relative priorities, and communicate the aggregate picture to elected officials. Each incident still has its own IC running its own response; Area Command coordinates between them, not within them.
What this looked like at Slack
At Slack, we adapted Area Command for situations where multiple incidents collided and needed coordination above the level of the individual ICs. The Area Command was itself an incident, with its own incident number, its own channel, and its own IC (the “Area Commander,” typically myself or another senior member of the incident management program). The Area Commander would interface with the IC from each active incident (or the IC would name a liaison) and focus on the questions that no individual IC could answer alone: which incident should get priority for deploying fixes, how to consolidate conversations with executive leadership, and whether a scarce resource should stay on one incident or move to another.
These are sacrifice decisions at the company-wide level: deliberately accepting a worse outcome on one incident to get a better outcome on another. The Area Commander would frame the decision for executives when needed, then work with the individual incidents to implement whatever was decided.
What made this work was keeping it lean and temporary. There was no Tech Lead at the Area Command level, because the technical work was happening within the individual incidents, not at the coordination level. If an IC designated a liaison to Area Command rather than handling that themselves, it was never the Tech Lead. The TL’s attention stayed on their incident’s technical workstreams.
We activated Area Command roughly twice a year, typically only for about an hour. It spun up when incidents started tripping over each other, and spun down as soon as the contention was resolved (often well before the incidents themselves were resolved). It’s not a standing organization; it’s a coordination pattern you reach for when you need it and put away when you don’t.
A rare but high-stakes situation
Most companies won’t need Area Command often. But the situations where it’s needed are precisely the situations where you don’t want to be figuring it out from scratch.
The pattern itself is simple: keep the individual incidents running independently, stand up a lightweight coordinator above them, and give that coordinator the authority and information to make the tradeoff calls that no individual IC can make. It doesn’t require elaborate tooling or training. It requires someone senior enough to make prioritization and resource allocation decisions, a channel for the coordination conversation, and a line of communication with each active incident.
The harder part is recognizing when simultaneous incidents have crossed from “running at the same time” into “interfering with each other.” Most companies stay in the first category most of the time. But any company with enough services and enough engineers will eventually hit the second, and having a name for the coordination pattern (and having thought through how it would work) makes the difference between a structured response and several ICs each lobbying for their own incident with no one positioned to make the call.
Area Command is one of the patterns I cover in my forthcoming book, “Incident Management for DevOps and SRE.” If you want to know when it’s available, sign up at im4ds.com.
If your company needs help building or improving your incident management practice, that’s the focus of my consulting at Great Circle.
by Brent Chapman | Jun 23, 2026 | Uncategorized
Most companies’ incident processes work well for routine incidents. An engineer gets paged, a few colleagues join the response, someone takes on the incident commander (IC) role, and the team works the problem together in a Slack channel. The IC can keep track of what everyone is doing because “everyone” is five or six active responders. Communication flows naturally. The process feels lightweight because it is.
The first scaling step: naming a tech lead
The first time this setup gets strained is usually when the IC has too many things competing for their attention. They’re trying to guide the technical investigation while also fielding questions from stakeholders, coordinating communication, and keeping the response organized. The IC’s attention fragments, and both the technical work and the coordination suffer.
The answer at this stage is naming a tech lead (TL) for the incident: someone who takes over direct management of the technical work while the IC handles everything outward-facing. Together, an IC and TL can effectively coordinate more responders than either could alone, because they’ve split the two biggest demands on attention into separate roles. Adding a TL helps you scale beyond what the IC alone could handle, but it only takes you so far.
When even IC and TL aren’t enough
When an incident has enough active responders working on different aspects of the problem simultaneously, even an IC and TL working together can’t directly coordinate all of them. The single channel becomes congested with interleaved conversations about different problem areas. Responders spend more time trying to follow the scroll than working the problem. People start working at cross purposes because nobody has visibility into what other responders are doing.
To be clear: it’s fine to have dozens of people watching an incident channel. Spectators following along for situational awareness is a sign of healthy transparency, not a coordination problem. The challenge is the active responders, the people actually working the incident, and what happens when there are more of them than an IC and TL can directly coordinate.
What experienced responders do next
In many companies, the experienced responders sense this breakdown before anyone names it. They’ve seen it before. They get frustrated with the chaos, quietly break off into smaller groups, and start working the problem in DMs, side huddles, or breakout channels where they can actually focus.
What they’re doing, whether they’d use the word or not, is forming workstreams: small teams organized around specific aspects of the problem. For example, in an incident involving data corruption across multiple systems, one workstream might focus on stopping the corruption, another on assessing the blast radius, and a third on coordinating customer communication. Each workstream has a natural focus and can work without wading through the noise of the other workstreams’ conversations.
This instinct is sound. The main channel has become too congested for focused technical work, and breaking into smaller, objective-focused workstream groups is exactly the right response. But when it happens informally, it creates its own problems. The IC and TL may not know the workstreams have formed. There’s no explicit coordination between the workstreams. Information that matters to one workstream gets trapped in another workstream’s side conversation. And at its worst, it shades into freelancing: experienced people working on what they think is most important, outside the IC’s and TL’s awareness, with nobody having the big picture across the effort. The whole thing only works when those particular experienced people happen to be responding. When they’re not, the incident stays in the chaotic single-channel mode, and the response suffers for it.
Making the instinct explicit
The fix is to formalize what experienced responders already do naturally. When an incident grows beyond what an IC and TL can directly coordinate (roughly eight to ten active responders, in my experience), break the response into named workstreams, each with a designated workstream lead who coordinates the work within that workstream.
The TL’s role shifts at this point. Instead of directly managing all the technical responders, the TL coordinates across workstream leads: connecting dots between workstreams, making sure one workstream’s fix isn’t creating problems for another, and maintaining the technical big picture that no single workstream can see. The IC continues to handle organizational coordination, stakeholder communication, and the overall direction of the response. It’s the same IC/TL partnership, scaled up one level.
One detail matters more than it might seem: name workstreams after what they’re trying to accomplish, not which team the people came from. “Stop the data corruption” is a workstream objective. “Database team” is an org chart label. In a complex incident, the work rarely falls along team boundaries; the whole point of the incident response program is to be able to form an ad hoc team to address the problems. A workstream focused on stopping data corruption might need engineers from the database team, the networking team, and the application team all working together. Naming by objective keeps everyone oriented toward the same goal and avoids the trap of siloing along reporting lines when the problem doesn’t respect those lines.
How fire departments scale their coordination
Fire departments face the same scaling challenge, and the way they handle it is instructive. The Incident Command System (ICS) explicitly defines how the organizational structure scales with the size of the incident. A single-engine response to a dumpster fire has one officer managing a 3-4 person crew. A multi-alarm apartment building fire has an IC at the top of an org chart spanning multiple engine and truck companies, rescue teams, ambulances, and other specialized units organized across multiple floors and faces of the building. As units arrive to join the response, the organizational structure scales up in well-defined ways to absorb them, and firefighters train for these transitions before they ever face a live fire.
The specific structure from ICS doesn’t translate directly to software incidents (we don’t need strike team leaders, for example). But the principle does: the coordination model must evolve in predictable, preplanned ways, and people need to practice making those changes before they’re in the middle of a crisis.
Getting started with workstreams
Most companies design their incident process for the incidents they have most often, and that’s reasonable. Routine incidents with a handful of responders don’t need workstream coordination. But if you’ve ever had an incident where the response felt scattered and chaotic, where experienced people quietly broke away to work independently, and where nobody had the big picture across all the parallel efforts (or even knew what all those efforts were), the issue was probably that your coordination model didn’t scale with the incident.
The shift to workstreams doesn’t happen all at once in an incident. When the IC or TL recognizes that the response has outgrown direct coordination, they might start by spinning up a single workstream for the most clearly defined problem area, while other responders continue working in the main channel under the TL’s direct coordination. As more distinct problem areas emerge, more workstreams form. It’s a gradual transition, not an instant cutover. And the reverse applies as well: as workstreams accomplish their objectives, they can be dissolved and their responders absorbed into the remaining workstreams or stood down. The coordination structure should contract as the incident winds down, just as it expanded when the incident grew.
A few practical starting points: when you spin up a workstream, give it its own channel so the work is visible, not DMs or side huddles where it disappears from the IC’s and TL’s view. Name it by its objective. Each workstream should have an identified lead who coordinates the work within the workstream and reports back to the TL. The TL connects the dots and coordinates across workstreams, making sure one workstream’s approach isn’t undermining another’s. And just like with Slack threads, key discoveries and decisions made within a workstream need to get shared back to the main channel immediately, so the IC, the TL, and other workstreams all have the full picture.
There’s more to managing workstreams well than a blog post can cover: how workstream leads communicate status, how the TL balances attention across workstreams, how to handle responders who need to move between workstreams as the situation evolves. I cover these in a chapter on large, long-running, and other special-situation incidents in my forthcoming book, Incident Management for DevOps and SRE. Sign up for publication updates at im4ds.com. If your company needs help building this into your incident process right now, my consulting practice is greatcircle.com/im.
The experienced responders on your team already instinctively self-organize when an incident gets big. Your process should be supporting and channeling that instinct into something reliable and repeatable, not leaving it to chance.
by Brent Chapman | Jun 16, 2026 | Uncategorized
Every large software development organization has made the same bargain: if nobody has to understand the whole system, we can build a more capable system than we otherwise could, even though it grows bigger and more complex. We break systems into components with well-defined interfaces so that each team needs to understand only the pieces it owns, plus the interfaces of its neighbors. That’s the point of every decomposition strategy, whether it’s microservices, bounded contexts, service ownership, or a well-modularized monolith. The architecture deliberately limits what any one person has to hold in their head.
It’s a sound strategy. It’s also why some incidents are so much harder than others.
Lorin Hochstein named this pattern beautifully in a recent post, The demon of the gaps. Failures that stay inside a single component are the easy ones; you page the owning team, they figure out what’s wrong, and they fix it. The hairy incidents emerge from unexpected interactions across components: several services throwing errors at once, or no services throwing errors while customers see broken behavior anyway. As Lorin puts it, “you’ve built an analysis solution but you’re now faced with a synthesis problem.” In order to scale, the architecture deliberately optimized away the need for whole-system understanding; now the whole system isn’t working, but nobody has that understanding to call on.
I’ve watched this play out in incident channels many times. Subject matter experts from six different teams, each reporting that their own service looks healthy. Six dashboards are green, but meanwhile, checkout is still failing for customers. The knowledge needed to explain what’s happening exists, distributed across six heads, but nobody is assembling the pieces. The responders need to understand how the system as a whole is behaving right now. That understanding has to be built live, under pressure, from multiple partial models. That’s synthesis work, and it doesn’t happen on its own.
Lorin observes that guidance on preparing for this work is almost nonexistent. Here’s the encouraging part: closing the structural gap is fairly straightforward. Most companies already use structured incident roles (incident commander, subject matter expert, customer liaison, etc.); they need to add a synthesis role, activated when needed for complex incidents.
Synthesis is a job. Name it.
The incident commander (IC) coordinates the overall response; every incident will have one. On the most complex incidents, though, where you need this synthesis function most, the trick is to also activate an incident tech lead (TL) to lead the technical investigation. The role is analogous to the tech lead role many teams have in their everyday structure, but its scope is the incident rather than one particular service. Most companies have never established the incident TL role, and for routine incidents they don’t miss it: the IC can handle the technical side along with everything else, but for complex incidents, the TL role can be incredibly valuable.
The incident TL job, properly understood, is the synthesis job: connecting observations across component boundaries, correlating the partial models from different subject matter experts, and maintaining the evolving picture of how the system is failing and what we’re doing about it. The TL doesn’t need to be the deepest expert in any single component. They need to be good at building a working model out of other people’s expertise, and much of that work is cross-checking, holding indications from different components up against each other and noticing the discrepancies: “If we’re seeing this in component A, we should be seeing that in component B, but we aren’t; why not?” “If A is doing this and B is doing that, the problem must be upstream of both.” “Wait, A says one thing but B says another; they can’t both be right, can they?”
The separation between IC and TL exists to protect that work, and it cuts both ways. Synthesis requires sustained, heads-down attention; you can’t reconstruct a system model in the gaps between stakeholder updates and staffing decisions. And the same complexity that makes an incident demand serious synthesis also multiplies the outward-facing work: more stakeholders to update, more escalations, more decisions about the response itself. The two loads peak together, and one person can’t carry both.
The IC takes everything outward-facing precisely so the TL can stay immersed in the technical picture, and the TL handles the heads-down focused work so that the IC has time for everything else. When I’m the incident commander, one of the most valuable things I can do for my tech lead is keep everyone else out of their hair. But the separation is a division of labor, not a wall. I like to think of the IC and the TL standing back to back, facing opposite directions, talking over their shoulders to keep each other informed. Each is watching a different part of the horizon, and together they have the whole picture.
The response team crosses the boundaries on purpose
An incident response is a temporary organization: an ad hoc team assembled across ownership boundaries for exactly as long as the incident lasts. Conway’s law observes that systems end up mirroring the communication structures of the organizations that build them, and the mirror works in both directions: your team boundaries and your component boundaries align, which is exactly what you want for everyday work. The incident structure deliberately cuts across those boundaries, because the gaps between components are where the problem lives. Pulling six SMEs into one channel isn’t enough by itself, though. A group of experts in the same room is a meeting; a group of experts with someone responsible for synthesizing what they know is a response.
The communication mechanisms are synthesis tools
The standard incident communication practices may seem like bureaucratic overhead until you see what they’re for. “Going around the horn” (each responder, in turn, briefly reports what they’re seeing and doing) forces the partial models into the open, where the TL can correlate them. A periodic situation report, or SitRep, forces someone to compress the current understanding into a few sentences; writing it is itself an act of synthesis, and reading it gives every responder the same baseline picture to work from. Narrating before you act keeps each responder’s local view visible to the whole room. None of these mechanisms exists for discipline’s sake. They’re how a group of people, each holding a partial model, builds and maintains a shared one.
Wildfires don’t respect organizational boundaries either
As is often the case in incident management, we can look to fire departments for inspiration and solutions. Consider a major wildfire. Dozens of agencies converge: federal, state, tribal, and local, some from hundreds or even thousands of miles away. No single agency understands the whole incident, with its terrain, weather, fuel, crews, and aircraft. The Incident Command System (ICS), the standard structure for emergency response in the US and beyond, is how all these disparate parts get pulled together into a coherent whole. ICS treats building the shared picture as a staffed function: a planning section tracks the situation and the resources, assembles the common operating picture, and distributes it to every responder through the incident action plan. Nobody simply hopes that shared understanding will emerge; somebody owns producing it.
Software companies can borrow that lesson directly: treat synthesis as a named responsibility rather than an emergent property. If the IC role at your company is defined as “project manager of the outage” and nobody is explicitly responsible for assembling the technical picture, the synthesis function is unowned, and it will show in your cross-boundary incidents. Establish the incident tech lead role. Protect it from outward-facing distraction. And practice it: when you run game days or tabletop exercises, choose scenarios that cross team boundaries, because those are the scenarios that exercise synthesis rather than component expertise.
Decomposition made whole-system understanding nobody’s everyday job, and that’s fine; it’s a good strategy with a known cost. Incident management structure is how you pay that cost only when you must, with machinery built for the moment.
I’m writing a book, “Incident Management for DevOps and SRE.” Sign up at im4ds.com to be notified when it’s available, and to get occasional progress updates and early access to selected content.
If your company needs help with incident management right now, that’s the focus of my consulting practice at GreatCircle.com/im.
by Brent Chapman | Jun 9, 2026 | Uncategorized
If your engineers are using AI coding assistants, your team is almost certainly shipping more code than they were before adopting these tools. That’s not surprising: the whole point of these tools is to accelerate how fast code moves from idea to production. The velocity story is real, and it’s the story most companies focus on.
The key question is, when that new code breaks in production at 3am, how well can the on-call engineers debug it?
The understanding gap
I’ve written before about how AI tools are quietly thinning the understanding that teams have of their own systems. The short version: AI-assisted development shifts how code gets produced in ways that leave the team with shallower collective knowledge of the codebase. Not because anyone is doing something wrong. Good teams still do design reviews, still do code review, still write documentation.
But when AI generates code, the team reviews the output rather than participating in the implementation choices. The understanding they build is real, but it’s not as deep as what they’d have if they’d built it together. TR Jordan of Tern captures the shift well: the old deal was that if it was worth your time to write the code, it was worth my time to read it. When the code is AI-generated, there’s so much more code to review that the deal breaks down, and the knowledge-sharing that used to be baked into the process has to be rebuilt deliberately.
During normal operations, that’s fine. Teams have time to read through unfamiliar code, query the AI, run experiments, consult documentation. The pace is forgiving.
When thinner understanding meets time pressure
The pager goes off at 3am, and within minutes the response becomes a team effort: the on-call engineer pulls in teammates, the incident tech lead drives the investigation, subject matter experts get paged. But the team’s effectiveness under pressure depends on their collective understanding of the systems and code involved. That understanding is exactly what’s gotten thinner as the code volume has increased and more of the codebase has been shaped by AI.
This doesn’t mean the team is helpless. They can still read the code, still query the AI about what it does, still use their debugging tools. But there’s a difference between understanding code well enough to work with it during the normal course of development and understanding it well enough to reason about its failure modes at 3am, under time pressure, with customers affected. The first is a comfortable margin. The second is where gaps in understanding become visible.
The more of the codebase that’s been shaped by AI, the more the incident response team is working in territory they know less deeply than they would have if they’d built it all themselves. Each individual piece of AI-generated code might be fine. But in aggregate, the team’s ratio of “code in production” to “code we understand deeply enough to debug under pressure” has shifted. And it’s shifted in the wrong direction for incident response.
From valuable to essential
Firefighters deal with a version of this problem every time they respond to a fire in a building they’ve never been inside. They don’t know the floor plan, the hazards, or the building’s history. What they rely on instead are general diagnostic skills: understanding building types and construction methods, knowing how fire behaves, reading smoke conditions and other indicators. They’ve trained specifically for navigating the unfamiliar, because in their line of work, the unfamiliar is the norm. And they don’t just rely on those skills in the moment. Between calls, they prepare: conducting familiarization visits to buildings in their district, having informal “what if?” discussions over the kitchen table, running whiteboard sessions, reviewing and updating pre-incident plans. They build as much understanding as they can before the alarm sounds, knowing it won’t be complete but also knowing that every bit of preparation helps.
The Google SRE book describes an analogous training approach for software engineers: building the general skill of dropping into an unfamiliar system under pressure. Using diagnostic tools and debugging surfaces. Following requests across service boundaries. Drawing inferences from logs and metrics. Making that process reflexive enough to work when the stakes are high and the clock is running.
That skill set has always been valuable, but AI-assisted development makes it essential. When a growing share of your production code was written or substantially shaped by AI, the ability to debug systems you didn’t build is no longer just a nice-to-have that distinguished your strongest engineers; it’s a core competency your entire on-call team needs.
Of course, this assumes you’ve invested in the infrastructure to support those skills: diagnostic tooling, distributed tracing, structured logging, debugging surfaces that actually reveal what’s happening across service boundaries. If your company is shipping more AI-generated code, the case for investing in observability infrastructure gets stronger, not weaker. The skills and the tooling go together.
What this means for your company
If your company is adopting AI coding tools, the question isn’t whether the understanding gap exists. It’s whether your incident management practices account for it.
Invest in general diagnostic skills. Don’t just train engineers on specific systems; train them to navigate unfamiliar ones. Structured debugging exercises, shadowing across teams, and practice with diagnostic tooling all build the kind of transferable skill that matters most when the code is unfamiliar.
Don’t assume familiarity will come from the work itself. When teams hand-wrote most of their code, system understanding was a natural byproduct of the development process. AI-assisted development weakens that link. Companies need to explicitly invest in building the shared understanding that used to come for free. Some of that investment is formal: structured on-call ramp-up, cross-team shadowing, and light-weight training exercises. But some of it is informal, and just as important: engineers walking each other through recent changes, pairing on debugging sessions, having “what would we do if X broke?” conversations over lunch.
Build understanding between incidents. Firefighters build a lot of their knowledge around the kitchen table between calls. Software teams need the equivalent, and they need to protect the time for it. Dedicate a regular slot in your weekly team meetings for disaster role-playing or system walkthroughs. Google’s SRE teams have done this for years with a practice they call “Wheel of Misfortune”. The key is, it’s not a big-deal formal exercise, it’s just how they spend the last ten minutes of a weekly meeting.
Treat this as an organizational capability problem. Adopting AI coding tools for velocity gains is an organizational decision. So is investing in the operational readiness to match. That’s not an argument against AI tools; it’s an argument for thinking about the full picture. Shipping more and faster is valuable. But the cost shows up at 3am, when code your team doesn’t fully understand breaks in production and the clock starts running.
I’m writing a book on incident management for DevOps and SRE that covers this and much more. Sign up at im4ds.com to be notified when it’s available.
If your company needs help preventing, preparing for, responding to, and learning from incidents, my consulting practice is greatcircle.com/im.
by Brent Chapman | Jun 4, 2026 | Uncategorized
AI is automating operational work at an accelerating pace. AI ops tools handle monitoring, remediation, environment management, and infrastructure tasks that engineers used to do themselves. The “AI SRE” product category barely existed two years ago; now every vendor in the space has one. AI-assisted development tools generate code, suggest architectures, and handle implementation details. These tools deliver real value; they’re saving teams real time on real work today.
But there’s a second-order effect that most companies deploying these tools aren’t accounting for: the operational work that AI is taking over was also how engineers unconsciously built the system knowledge they need during incidents.
The engineer who regularly works with the infrastructure (designing, deploying, scaling, troubleshooting, tuning, investigating when things look wrong) develops an intimate knowledge of the systems: the dependency chains, the failure modes, what “healthy” looks like.
Every generation of automation has eroded some of that knowledge, and that’s usually been a worthwhile trade. Auto-scaling is a good example: it works so well that nobody thinks about scaling behavior day-to-day. Right up until the auto-scaler walks off a cliff, spawning so many new frontends that they overwhelm the database backend with connection requests and cache warmups, then time out and abort before getting online, wasting all the work the backend did to try to start them, and kicking off a crash loop of attempting to start, timing out, and retrying. The engineers responding to that incident need to understand scaling dynamics that haven’t been part of anyone’s daily awareness since the auto-scaler took over. That pattern predates AI entirely.
But AI is automating a broader range of operational work, faster, and the knowledge that erodes with it is correspondingly deeper.
Here’s why that matters for incidents: incidents are, by definition, the situations that the automation can’t handle. They’re what’s left over when everything that could be automated has been. And as the automation (both traditional and AI) gets more capable, the left-overs get messier and more complicated. The people who need to respond to those situations are the same people whose day-to-day work is increasingly mediated by AI. They have less deep understanding of the systems they’re being asked to debug, investigate, and reason about under pressure.
This is true even if you aren’t using any AI tools during incident response itself. The awareness erosion happened before the incident started.
Part of why this is happening so fast is that many companies already viewed operational work as lower-value toil, ripe for automation. Fred Hebert pointed out a revealing asymmetry in how AI tools are marketed: coding assistants are framed as augmenting the engineer (they’re “partners” and “teammates,” and the developer stays in control), while AI ops tools are framed as replacing the work entirely (“machines on-call for humans,” “stop firefighting, start innovating”). The framing reveals what the market thinks the work is worth: not much. If your company takes that view, that operational work is grunt work to be automated away, it’s going to underinvest in the human knowledge that effective incident response requires.
The operational work wasn’t just toil; it was keeping people’s heads in the game. Situational awareness gets built as a side effect of doing the work; you don’t realize its value until the work goes away and you discover the hard way that the awareness went with it.
The ironies of automation
In 1983, cognitive psychologist Lisanne Bainbridge published a paper called “Ironies of Automation” that described a paradox: the more you automate a process, the less aware the human operator is of the system’s current state, and the harder it becomes for them to handle the situations that the automation can’t. The most striking example is commercial aviation. Autopilot systems handle routine flight so well that pilots spend less time actively engaged with what the aircraft is doing. At the same time, pilots’ skills atrophy from disuse, since the autopilot is handling more and more of the work of flying. But when the autopilot fails or encounters something it can’t handle, the pilot needs to take over in exactly the kind of unusual situation that demands the most current awareness of the aircraft’s state and the most skill in responding to it.
There’s an old pilot joke that the scariest words you can hear in the cockpit are “Huh? What’s it doing now?” It’s funny because it captures exactly the gap Bainbridge described: the crew has lost track of what the automation is doing, at the moment when they need to understand it most.
This creates a double bind: the pilot is less aware of what’s happening right now, and over time, less practiced at handling it. The gap widens from both directions.
John Allspaw brought this concept to the DevOps and SRE community through his influential “A Mature Role for Automation” blog series, and it’s been shaping how we think about automation in software operations ever since. The principle isn’t anti-automation; it’s a caution about what automation displaces, and about what you need to do to compensate.
This is happening right now, fast
That pattern is now playing out with AI across operations, with one critical difference: the tempo. In aviation, the ironies of automation emerged over decades as new aircraft and autopilot systems were gradually introduced. In tech, traditional automation has been gradually eroding hands-on system knowledge for years. AI is compressing that same dynamic into months and weeks, because AI capabilities are advancing faster than any previous generation of automation, and because AI is automating categories of work that previous tooling couldn’t touch.
Consider an engineer whose team recently adopted AI tools for infrastructure management and code generation. Six months ago, they knew their deployment pipeline intimately because they built it, tuned it, and fixed it when it broke. They knew which services were fragile because they’d spent time troubleshooting them. They had a mental model of the system’s architecture because they’d worked with it directly. Now AI handles much of that work. The engineer is more productive. But when something goes wrong that the AI can’t resolve (when the situation becomes an incident), the engineer’s mental model is fuzzy and possibly six months stale. The discrepancy they would have noticed because they’d just been troubleshooting that service last week is now invisible to them.
And if you’re also using AI tools during incident response (as many companies are beginning to, for sitrep drafting, channel summarization, log analysis, and the like), the problem compounds: less system knowledge being brought into the incident, less situational awareness during the response itself.
Over time, the skills atrophy too. Engineers who rarely troubleshoot manually get worse at structured debugging. Engineers who rarely investigate anomalies lose the intuition for what’s worth pursuing. The immediate loss of awareness compounds into a longer-term erosion of skill, and as AI handles a wider range of situations, the situations that still require human judgment become harder and rarer. The humans facing those situations need to be more capable than before, not less.
What to do about it
None of this means companies should stop using AI ops tools. The productivity gains are real, and the direction is clear. The point is to deploy these tools with your eyes open about what the automation displaces, and to invest deliberately in maintaining the knowledge and skills that the automated work used to build.
The aviation industry recognized this problem decades ago. Their solution: frequent mandatory recurrent training, specifically designed to keep pilots practiced on the skills that routine automated flight no longer exercises. The tech equivalent is exercises, game days, and deliberate hands-on work with the systems your team is responsible for. Companies like Uptime Labs are starting to build tools for exactly this kind of recurrent training.
Keep engineers connected to the systems they’re responsible for. If AI handles most of the day-to-day operational work, create deliberate opportunities for engineers to work with the systems directly: manual deployments during low-risk windows, hands-on troubleshooting during exercises, periodic deep-dives into the infrastructure that go beyond what the AI dashboards show.
Treat AI tools as useful but not required. Build your processes so that an AI tool outage is an inconvenience, not a crisis. The engineers who can still function without the tools are the ones you’ll need when the tools aren’t available (and they won’t be, eventually; tools fail, sometimes during the incidents where you need them most).
Run exercises and game days that test system understanding, not just process compliance. A tabletop exercise where the scenario is “your AI ops tools are down and you need to investigate a production issue manually” will tell you a lot about how much system knowledge your team has actually retained.
The bottom line
The ironies of automation aren’t an argument against automation. Bainbridge wasn’t arguing against autopilots, and this isn’t an argument against AI ops tools. The argument is that automation changes the human’s relationship to the work in ways that are easy to miss. The operational work wasn’t just toil; it was building the understanding that people need when things go wrong. When AI takes over that work, the understanding erodes, quietly and steadily, until the next incident reveals how much has been lost. Companies that deploy AI ops tools without accounting for this will discover the gap at the worst possible moment: during the incident that the AI can’t handle, when the engineers discover they’re no longer ready to handle it either.
I’m writing a book on incident management for DevOps and SRE that covers this and much more. Sign up at im4ds.com to be notified when it’s available.
If your company needs help with incident management right now, my consulting practice is greatcircle.com/im.
Recent Comments