Picture the post mortem.

A company has just disclosed that one of its AI agents did something expensive. Money moved that should not have, or data left that should not have, or a system was changed in a way that took days to undo. The incident review convenes, and the engineers go looking for the bug.

They do not find one.

The model performed exactly as designed. Given the inputs it received, its reasoning was sound and its output was the kind of output it was built to produce. The APIs it called returned success, because the calls were well formed and the agent was authenticated and authorised to make them. The credentials were valid and unexpired. The tools worked. The logs captured everything in clean, complete detail. Every component, examined in isolation, did precisely what it was supposed to do.

And the outcome was still a disaster.

This is the agent incident that the industry is not ready for, because it does not look like the incidents we know how to prevent. There is no vulnerability to patch, no bug to fix, no negligent engineer to blame. The agent did nothing wrong. Every part worked. And that is exactly the problem.

Correctness Is Not Safety

There is a principle from the study of complex systems that the AI agent world has not yet absorbed, and it is the key to the whole thing. The correctness of the individual components does not give you the safety of the system. These are different properties, and the second does not follow from the first.

A component is correct when it does what its specification says it should do. The model is correct when it produces reasonable output for its input. The API is correct when it executes well formed authorised requests. The auth layer is correct when it admits valid credentials and rejects invalid ones. Each of these can be perfect, provably, individually correct, and the system they compose can still produce catastrophic outcomes, because safety is not a property of any component. It is a property of their interaction.

Engineers know this in other domains. A safety review of an aircraft or a chemical plant does not stop at confirming each part meets its spec. It asks what happens at the boundaries, in the interactions, in the sequences the individual specifications never contemplated. The discipline of system safety exists precisely because correct components routinely combine into unsafe systems. The famous industrial accidents were almost never a single broken part. They were correct parts interacting in a way no one governed.

AI agents are being built as if component correctness were enough. Make the model better. Secure the credentials. Harden the APIs. Each effort is worthwhile and each makes its component more correct. None of them addresses safety, because safety lives in the space between the components, in the actions that cross from one to the next, and almost nothing in the current agent stack governs that space.

The entire investment is going into the left column. The disaster comes from the right.

Why Agents Make This Acute

Component correctness has always been insufficient for safety, but you could get surprisingly far on it when systems were simpler. Agents remove that grace, for reasons specific to what they are.

A traditional system has a fixed set of interactions. The components connect in ways the designers chose, and the paths between them are enumerable. You can, with effort, reason about the interaction space, because it is bounded. Most of the time the correct components interact in the small number of ways the design intended, and careful engineering covers the edges.

An agent has an open interaction space. It decides at runtime which components to invoke, in which order, with which data, based on model reasoning over a context that includes things the designers never saw. The number of ways the correct components can be made to interact is not bounded by the design. It is bounded only by what the agent can be led to do, and the agent can be led to do a great deal, because its behaviour is shaped by data, some of which an adversary may control.

This is why the disaster needs no broken part. The attacker, or simply an unlucky combination of inputs, does not need to break the model or steal a credential or find a bug in the API. They need only to arrange for the correct components to interact in a way that is individually valid at every step and collectively catastrophic. Each action passes its local check. The model's output is reasonable for its input. The API call is well formed and authorised. The sequence is the problem, and no component sees the sequence.

   the disaster is assembled from correct steps

   step 1   model produces reasonable output      ✓ correct
   step 2   agent calls an API it is authorised on  ✓ correct
   step 3   data returned, agent reasons over it     ✓ correct
   step 4   agent calls another authorised API       ✓ correct
   step 5   action executes on valid credentials     ✓ correct
            ────────────────────────────────────────
            every step valid · the outcome a breach

Each row would pass its own review. The breach is in the column that does not exist on the right: the one that asks whether this sequence, as a whole, in this context, should be allowed to happen. Nothing in a stack of correct components asks that question, because it is not a question any single component is positioned to ask.

Anatomy of a Blameless Disaster

Make it concrete, because the abstraction hides how ordinary each step looks. A company runs an agent that handles customer operations. It reads incoming requests, looks up account details, and can issue credits, update records, and send confirmations. It has been in production for months without incident. Every component has a clean track record.

A request arrives. It is unusual but not obviously malicious, the kind of edge case that shows up a few times a week. Embedded in it, in ordinary language, is content that reframes the customer's situation in a way that leads the agent, reasoning correctly over what it was given, to conclude that a large credit is warranted and that the account contact details should be updated to match the request.

Follow the steps. The model reads the request and produces a conclusion that is reasonable given the text it was handed; a human reading the same crafted text might lean the same way. Correct. The agent looks up the account, an authorised read it performs constantly. Correct. It issues the credit through the credits API, well formed, within its scope, authenticated. Correct. It updates the contact details, another authorised write. Correct. It sends a confirmation, exactly as designed. Correct. Five steps, five green checks, and the company has just issued a large credit and handed account control to whoever crafted the request.

Now run the post mortem. The model did not malfunction; its reasoning was sound for its input. The credits API did not malfunction; it executed an authorised, well formed call. The auth layer did not malfunction; every action was taken by a valid, correctly scoped identity. The logs are complete; they show every step in order. There is no bug ticket to file. There is no component to fix. There is no engineer who failed to do their job. The review will conclude, accurately, that everything worked as designed, and that conclusion will be the most alarming sentence in the document, because it means the thing that failed was not in the inventory of things that could fail.

What was missing was a control that looked at the sequence rather than the steps. A large credit triggered by an inbound request, immediately followed by a change of account contact details, is a pattern. No single component could see the pattern, because each component saw only its own step. The pattern is exactly what an authorisation layer evaluating each action in context would have caught, and exactly what a stack of individually correct components cannot.

The Three Reassurances That Will Fail

When the disaster comes, three reassurances that enterprises currently rely on will turn out to have been answering the wrong question. It is worth naming them now, because each one feels like safety and is actually correctness in disguise.

"The model is state of the art." A better model is a more correct component. It reasons better, hallucinates less, follows instructions more reliably. Every one of those is an improvement in component correctness, and none of them is safety. A more capable model that is led into a harmful action sequence executes it more competently. Capability is not the same axis as governance. You can have all the capability in the world and no control over what it is allowed to do.

"Everything is authenticated and access controlled." Authentication confirms identity. Access control confirms that an identity may reach a resource. Both are correctness properties of the access layer, and both can be flawless while the disaster proceeds, because the harmful actions were taken by a correctly authenticated identity that was correctly authorised to reach those resources. The access layer did its job. Its job was never to evaluate whether this particular action, in this particular sequence, should occur.

"We have full logs." Logging is the most seductive of the three, because after the incident the logs will be complete and detailed, and that completeness feels like control. But a complete log is a perfect record of a disaster that already happened. It is correctness of the observability layer, and observability is not prevention. The log will tell the post mortem, in flawless detail, the exact sequence of correct steps that produced the catastrophe. It will not have stopped a single one of them.

Three reassurances, three forms of correctness, and not one of them is the thing that was missing. The thing that was missing sits in a different place entirely.

The Missing Layer Is Between the Parts

If safety is not in the components, where is it. It is in the layer that governs the actions passing between them, and in most agent deployments that layer simply does not exist.

Consider where each action goes. The agent decides to do something, and that intention becomes a call that crosses from the agent into a real system. That crossing is the moment safety is determined, because that is where an intention becomes a consequence. Before the crossing, the agent has merely thought about acting. After it, the action has happened. The crossing is the only place where you can evaluate the action against everything that matters, what it is, what came before it, where its instruction originated, what policy applies, and decide whether it should proceed, while it is still a decision rather than a fact.

This is the layer that governs the space between the parts. It does not make the model more correct or the API more correct. It sits at the boundary every action must cross and asks the question no component asks: given the full context, should this specific action happen. It is an authorisation layer, and its entire purpose is to provide the system level safety that component level correctness cannot.

   without the layer                   with the layer

   model ─► API ─► system              model ─► [ authorise ] ─► API ─► system
   (each correct, no                            evaluate the action
    one governs the                             at the crossing,
    crossing)                                   block / allow / escalate
                                                ────────────────────────
   correct parts,                       safety enforced where
   unsafe system                        intention becomes consequence

Notice what this does to the disaster sequence from earlier. Each individual step was locally valid, and each would still pass its own component check. But now each step also crosses the authorisation layer, and the layer sees what no component could: that this step, in this sequence, with this provenance, against this policy, is not one that should proceed. The model is still correct. The API is still correct. The credentials are still valid. And the disaster does not happen, because the one thing that was missing, a control at the crossing, is now present. Safety was never going to come from making the parts more correct. It comes from governing what passes between them.

Why This Is the Hard Kind of Failure

The reason this matters now, urgently, is that this category of failure is the hardest kind for an organisation to defend against, on every axis except one.

It is the hardest to predict, because there is no defect to find in advance. You cannot scan for it, because nothing is broken. A security review that examines each component will pass every component, and conclude the system is sound, and be wrong, because the review looked at the parts and the danger is in the interactions.

It is the hardest to attribute, because after the fact there is no negligent party. The model team did their job. The platform team did theirs. The security team authenticated and authorised correctly. Everyone can show that their component met its specification, and they will all be right, and the disaster still happened. Blame has nowhere to land, which is uncomfortable, but the deeper problem is that without a place for responsibility there is no obvious owner for the fix.

It is the hardest to accept, because it contradicts a deeply held engineering intuition that if every part is correct, the system is correct. That intuition is wrong for complex interacting systems, and it has been known to be wrong for decades in other safety critical fields, but it is so intuitive that organisations will resist the conclusion even while standing in the wreckage of an incident that proves it.

And then there is the last row, which is the hopeful one. This failure is hard to predict, hard to attribute, and hard to accept, but it is not hard to prevent. The prevention is structural and it is known. You put a control at the crossing. You govern the actions that pass between the components, evaluating each one against context and policy before it executes. You do not have to predict the specific sequence, because the layer evaluates whatever sequence actually occurs. You do not have to make every component more correct, because you are no longer relying on component correctness for safety. The hardest failure to see coming is one of the most straightforward to design out, once you accept where it actually comes from.

The Sentence That Will Be in Every Post Mortem

Here is the line that will appear, in some form, in the incident reviews of the next few years, as enterprises work out what happened to agents they were sure they had secured.

"Everything worked as designed."

It will be written as an admission of confusion, because the people writing it will have started from the assumption that if everything worked, nothing bad could happen. They will have audited the model and found it sound. Audited the access controls and found them correct. Read the logs and found them complete. And they will be left staring at a sentence that should be reassuring and is instead the precise description of the failure.

Everything worked as designed is not the absence of a problem. For agentic systems, it is the signature of the specific problem this whole piece is about. It means every component was correct and the system was still unsafe, which means the safety was supposed to live somewhere that nobody built. It will read like a mystery and it is not one. It is the predictable result of investing entirely in component correctness and not at all in the layer that governs the actions between components.

The enterprises that internalise this before their incident, rather than after, will do one specific thing differently. They will stop treating the security of agents as a question of making each part more trustworthy, and start treating it as a question of governing what the parts are allowed to do when they act. They will put a control at the crossing. And when their post mortem says everything worked as designed, it will be describing a near miss that the authorisation layer caught and recorded, rather than a disaster that every correct component faithfully helped along.

The agent will do nothing wrong. The model will be right, the credentials valid, the APIs healthy, the logs complete. Insisting on the correctness of the parts was never the path to a safe system, and for agents it is not even close. Safety was always going to be a property of the whole, enforced in the space between the components, at the moment each action crosses from intention into consequence. That is the layer. It is the one that was missing, and it is the one that decides whether everything working as designed is a reassurance or an epitaph.

Xybern is the authorisation layer for enterprise AI agents. Every agent action is enforced, audited, and governed before it executes. Learn more at xybern.com or read the technical documentation at docs.xybern.com.