AI is automating operational work at an accelerating pace. AI ops tools handle monitoring, remediation, environment management, and infrastructure tasks that engineers used to do themselves. The “AI SRE” product category barely existed two years ago; now every vendor in the space has one. AI-assisted development tools generate code, suggest architectures, and handle implementation details. These tools deliver real value; they’re saving teams real time on real work today.

But there’s a second-order effect that most companies deploying these tools aren’t accounting for: the operational work that AI is taking over was also how engineers unconsciously built the system knowledge they need during incidents.

The engineer who regularly works with the infrastructure (designing, deploying, scaling, troubleshooting, tuning, investigating when things look wrong) develops an intimate knowledge of the systems: the dependency chains, the failure modes, what “healthy” looks like.

Every generation of automation has eroded some of that knowledge, and that’s usually been a worthwhile trade. Auto-scaling is a good example: it works so well that nobody thinks about scaling behavior day-to-day. Right up until the auto-scaler walks off a cliff, spawning so many new frontends that they overwhelm the database backend with connection requests and cache warmups, then time out and abort before getting online, wasting all the work the backend did to try to start them, and kicking off a crash loop of attempting to start, timing out, and retrying. The engineers responding to that incident need to understand scaling dynamics that haven’t been part of anyone’s daily awareness since the auto-scaler took over. That pattern predates AI entirely.

But AI is automating a broader range of operational work, faster, and the knowledge that erodes with it is correspondingly deeper.

Here’s why that matters for incidents: incidents are, by definition, the situations that the automation can’t handle. They’re what’s left over when everything that could be automated has been. And as the automation (both traditional and AI) gets more capable, the left-overs get messier and more complicated. The people who need to respond to those situations are the same people whose day-to-day work is increasingly mediated by AI. They have less deep understanding of the systems they’re being asked to debug, investigate, and reason about under pressure.

This is true even if you aren’t using any AI tools during incident response itself. The awareness erosion happened before the incident started.

Part of why this is happening so fast is that many companies already viewed operational work as lower-value toil, ripe for automation. Fred Hebert pointed out a revealing asymmetry in how AI tools are marketed: coding assistants are framed as augmenting the engineer (they’re “partners” and “teammates,” and the developer stays in control), while AI ops tools are framed as replacing the work entirely (“machines on-call for humans,” “stop firefighting, start innovating”). The framing reveals what the market thinks the work is worth: not much. If your company takes that view, that operational work is grunt work to be automated away, it’s going to underinvest in the human knowledge that effective incident response requires.

The operational work wasn’t just toil; it was keeping people’s heads in the game. Situational awareness gets built as a side effect of doing the work; you don’t realize its value until the work goes away and you discover the hard way that the awareness went with it.

The ironies of automation

In 1983, cognitive psychologist Lisanne Bainbridge published a paper called “Ironies of Automation” that described a paradox: the more you automate a process, the less aware the human operator is of the system’s current state, and the harder it becomes for them to handle the situations that the automation can’t. The most striking example is commercial aviation. Autopilot systems handle routine flight so well that pilots spend less time actively engaged with what the aircraft is doing. At the same time, pilots’ skills atrophy from disuse, since the autopilot is handling more and more of the work of flying. But when the autopilot fails or encounters something it can’t handle, the pilot needs to take over in exactly the kind of unusual situation that demands the most current awareness of the aircraft’s state and the most skill in responding to it.

There’s an old pilot joke that the scariest words you can hear in the cockpit are “Huh? What’s it doing now?” It’s funny because it captures exactly the gap Bainbridge described: the crew has lost track of what the automation is doing, at the moment when they need to understand it most.

This creates a double bind: the pilot is less aware of what’s happening right now, and over time, less practiced at handling it. The gap widens from both directions.

John Allspaw brought this concept to the DevOps and SRE community through his influential “A Mature Role for Automation” blog series, and it’s been shaping how we think about automation in software operations ever since. The principle isn’t anti-automation; it’s a caution about what automation displaces, and about what you need to do to compensate.

This is happening right now, fast

That pattern is now playing out with AI across operations, with one critical difference: the tempo. In aviation, the ironies of automation emerged over decades as new aircraft and autopilot systems were gradually introduced. In tech, traditional automation has been gradually eroding hands-on system knowledge for years. AI is compressing that same dynamic into months and weeks, because AI capabilities are advancing faster than any previous generation of automation, and because AI is automating categories of work that previous tooling couldn’t touch.

Consider an engineer whose team recently adopted AI tools for infrastructure management and code generation. Six months ago, they knew their deployment pipeline intimately because they built it, tuned it, and fixed it when it broke. They knew which services were fragile because they’d spent time troubleshooting them. They had a mental model of the system’s architecture because they’d worked with it directly. Now AI handles much of that work. The engineer is more productive. But when something goes wrong that the AI can’t resolve (when the situation becomes an incident), the engineer’s mental model is fuzzy and possibly six months stale. The discrepancy they would have noticed because they’d just been troubleshooting that service last week is now invisible to them.

And if you’re also using AI tools during incident response (as many companies are beginning to, for sitrep drafting, channel summarization, log analysis, and the like), the problem compounds: less system knowledge being brought into the incident, less situational awareness during the response itself.

Over time, the skills atrophy too. Engineers who rarely troubleshoot manually get worse at structured debugging. Engineers who rarely investigate anomalies lose the intuition for what’s worth pursuing. The immediate loss of awareness compounds into a longer-term erosion of skill, and as AI handles a wider range of situations, the situations that still require human judgment become harder and rarer. The humans facing those situations need to be more capable than before, not less.

What to do about it

None of this means companies should stop using AI ops tools. The productivity gains are real, and the direction is clear. The point is to deploy these tools with your eyes open about what the automation displaces, and to invest deliberately in maintaining the knowledge and skills that the automated work used to build.

The aviation industry recognized this problem decades ago. Their solution: frequent mandatory recurrent training, specifically designed to keep pilots practiced on the skills that routine automated flight no longer exercises. The tech equivalent is exercises, game days, and deliberate hands-on work with the systems your team is responsible for. Companies like Uptime Labs are starting to build tools for exactly this kind of recurrent training.

Keep engineers connected to the systems they’re responsible for. If AI handles most of the day-to-day operational work, create deliberate opportunities for engineers to work with the systems directly: manual deployments during low-risk windows, hands-on troubleshooting during exercises, periodic deep-dives into the infrastructure that go beyond what the AI dashboards show.

Treat AI tools as useful but not required. Build your processes so that an AI tool outage is an inconvenience, not a crisis. The engineers who can still function without the tools are the ones you’ll need when the tools aren’t available (and they won’t be, eventually; tools fail, sometimes during the incidents where you need them most).

Run exercises and game days that test system understanding, not just process compliance. A tabletop exercise where the scenario is “your AI ops tools are down and you need to investigate a production issue manually” will tell you a lot about how much system knowledge your team has actually retained.

The bottom line

The ironies of automation aren’t an argument against automation. Bainbridge wasn’t arguing against autopilots, and this isn’t an argument against AI ops tools. The argument is that automation changes the human’s relationship to the work in ways that are easy to miss. The operational work wasn’t just toil; it was building the understanding that people need when things go wrong. When AI takes over that work, the understanding erodes, quietly and steadily, until the next incident reveals how much has been lost. Companies that deploy AI ops tools without accounting for this will discover the gap at the worst possible moment: during the incident that the AI can’t handle, when the engineers discover they’re no longer ready to handle it either.

---

I’m writing a book on incident management for DevOps and SRE that covers this and much more. Sign up at im4ds.com to be notified when it’s available.

If your company needs help with incident management right now, my consulting practice is greatcircle.com/im.