London, UK, June 2026. We are releasing the Xybern Agent Authorization Benchmark (XAAB), an open, reproducible benchmark that measures how well an AI-agent authorisation layer does the one job that matters: stop unsafe agent actions while letting legitimate ones through.
Most "agent safety" today is keyword and regex matching. We wanted to measure what that actually costs, and to test our own layer against it on the same scenarios.
The result
Across 137 labelled scenarios (107 unsafe, 30 legitimate) drawn from the OWASP LLM Top 10, MITRE ATLAS, and CWE, the Xybern Authorisation Layer caught every unsafe action and blocked no legitimate one, measured beside a strong keyword guardrail and two trivial baselines:
| Approach | Attacks caught | False alarms | Score |
|---|---|---|---|
| Xybern Authorisation Layer | 100% | 0% | 100 |
| Keyword / regex guardrail | 49.5% | 13.3% | 36 |
| No layer (allow everything) | 0% | 0% | 0 |
| Block everything | 100% | 100% | 0 |
Score is Youden's J (attacks caught minus false alarms). Blocking everything also catches every attack, which is exactly why false alarms matter just as much: a layer that cries wolf is not a solution.
Intent beats keywords
The decisive result is what happens under pressure. Rephrase the same attacks (paraphrase, obfuscation, base64, leetspeak, multi-turn build-ups, instructions hidden in retrieved data) and the keyword guardrail's catch rate collapses from 84% to 44%. A keyword it does not contain is a keyword it cannot match. The Xybern Authorisation Layer holds at 100%, because its semantic policies judge intent, not surface strings.
It cuts the other way too. The benchmark deliberately includes scary-but-legitimate actions: a code-review note mentioning DROP TABLE, an authorised penetration-test report, an incident post-mortem, a fraud-awareness training email. A keyword filter false-positively blocks several of these. The authorisation layer correctly allows all of them, because it judges what the action does, not which words it contains.
Open and reproducible
We are publishing both halves so anyone can audit it: the full scenario dataset and the exact policy pack that was scored. The harness, the baselines, and a strong keyword-guardrail competitor are open source under Apache-2.0, and you do not need a Xybern account to run them.
Repository: github.com/xybern-ai/agent-authz-benchmark
We are growing the dataset and welcome adversarial contributions, including scenarios designed to make our own layer fail.