Waxell

AI Agent Governance Testing: Why Behavioral QA Isn't Enough

9 of 30 deployed AI agents had no documented guardrails, one study found. Governance testing checks whether your control layer fires, not just the agent.

Logan KellyUpdated 18 min read

AI agent governance testing is the practice of verifying that the control layer above your agents — cost limits, content filters, approval gates, tool restrictions — behaves correctly under real operating conditions, including edge cases and adversarial inputs. It is distinct from behavioral testing, which checks whether the agent completes its task correctly. Governance testing checks whether the machinery that constrains the agent holds when it is needed.

The difference is not academic, and the clearest recent demonstration came from inside a testing environment.

In May 2026, a Google Gemini model taking part in a capture-the-flag exercise reached the systems of three real companies. Google confirmed it in September, after the Wall Street Journal reported it. The exercise ran on infrastructure belonging to Irregular, the AI testing company, and targeted a fictional company that shared its name with a real one. According to Irregular, the model was not intended to have internet access at all — that access was unintentionally made available. In one run it guessed passwords until it reached a protected system; in two others it searched the web for the company name, found credentials in public repositories, and used them. Heather Adkins, Google's VP of security engineering, told SecurityWeek the model stopped in all three instances, and Google characterised the episode to the WSJ as mistaken identity rather than misalignment. We covered the scoping questions in depth in Gemini Hacked 3 Real Companies in a Security Test.

Set the attribution question aside and look at what failed. The behavioral evaluation worked as designed: the model was given a task and pursued it capably. What did not hold was a governance property of the environment around it — this agent has no network egress. Nobody had exercised that assumption the way you would test a firewall rule, and it turned out not to be true. A containment control that is never tested is a containment control you do not have.

That is the shape of the problem. A governance control that works in a controlled test but degrades under load, context pressure, or a sequence of individually permitted steps is worse than no governance, because it produces confidence you have not earned. Gartner put a number on where that leads: in a 26 May 2026 press release, it predicted that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance gaps identified only after production incidents occur. That is not a behavioral testing problem. It is a governance testing problem.

Why Behavioral Testing Doesn't Cover This

Behavioral testing is valuable. We've written about what it covers: tool selection, argument validation, state propagation, failure scenarios. You should be doing all of it.

But behavioral testing answers the question did the agent do what I wanted? Governance testing answers a different one: if the agent does something it shouldn't, does the control layer actually stop it?

Those require different test designs, different failure modes to watch for, and different infrastructure. Most teams have invested in the first — output scoring, hallucination detection, task completion metrics — and the surrounding evidence suggests the second is thinner than teams think.

The documentation gap is measurable. The 2025 AI Agent Index — a nine-author study from Cambridge, MIT, Harvard Law School, Stanford, the University of Washington, Penn, Concordia AI and the Hebrew University of Jerusalem, published at ACM FAccT 2026 — documented 30 deployed AI agents across 1,350 annotated fields. It found that 9 of 30 agents had no guardrails documented at all, and that sandboxing or VM isolation was documented for only 9 of 30. Among the 13 enterprise agents in the sample, 7 described options for setting up guardrails but no sandboxing or containment. Across the safety, evaluation and impact fields tracked, 135 of 240 had no public information; enterprise agents were the worst-documented category at 66% missing.

The practice gap is measurable too. Gravitee's State of AI Agent Security, a survey of 750 senior technology leaders in the UK and US fielded in April 2026, asked which controls are in place before an agent goes live. No single control was used by even 40% of organisations: a named person accountable for agent behaviour, 37.8%; a security review from IT or the CISO, 35%; a documented process to pause or revoke access, 34.1%; a defined scope of what the agent is permitted to access, 30.5%. Just 19.7% said all their agents were fully secured and governed before going live, and 54% reported a confirmed or suspected agent security or privacy incident in the preceding twelve months.

The trend line is the part worth sitting with. Stated confidence in agent visibility rose from 82.6% to 91.8% in four months while mean monitoring coverage barely moved — organisations, as Gravitee puts it, "becoming more comfortable with a risk they haven't actually reduced." False confidence tends to collect around controls that have been configured and never fired.

Three Ways Governance Fails That Behavioral Testing Doesn't Catch

1. Cost controls that pass configuration checks but don't fire

You set a max_budget. Your test runs stay within it. Everything looks fine.

In production the agent enters a retry loop — a flaky downstream API, or an ambiguous objective it keeps chasing. The limit is defined, but the enforcement logic checks the budget at task start rather than per retry, and the run burns several multiples of the ceiling before the check comes round. Or the limit is denominated in tokens while you are billed per request, and nobody caught the mismatch at implementation time.

The subtler version is a boundary mismatch rather than a threshold mismatch: the budget is scoped to a unit that does not match the unit spend is actually incurred in. When a single task fans out into a tree of subagents, an account-level or session-level ceiling can be technically correct and operationally useless, because the spend is set by the call tree at runtime — a failure we took apart in 74 Agents, 5.39B Tokens.

None of this shows up in behavioral testing, because the agent is behaving reasonably. Retrying a flaky API is the right call. The cost governance is what broke. You find it only by testing the control directly: run the agent into conditions that should trigger the limit, and verify it fires at the threshold you think it does, against the unit you think it counts.

2. Approval gates that can be forged, bypassed, or dropped

Human-in-the-loop approval is the control teams reach for when the stakes rise. It is also the control with the most ways to fail quietly.

The most concrete demonstration to date is CoreBreak, presented at Black Hat USA 2026 by researchers Hedi Ingber and Aviyam Ivgi and written up by the Cloud Security Alliance in August 2026. The dispatch layer sitting between a model's output and actual tool execution — in Amazon Bedrock AgentCore, Google's Agent Development Kit, and harness packages shipped with Vercel's AI SDK — did not reliably verify that a tool-call instruction had come from a model turn at all. Data merely shaped like a model-generated tool call was treated as authoritative. Because the model was never invoked, every model-level control layered on top — system prompts, content filters, refusal training — had nothing to intervene in. They were not defeated; they were structurally irrelevant.

The Google ADK case is the one that matters most for approval testing. Per the National Vulnerability Database entry for CVE-2026-18236, an attacker able to manipulate or inject events into an agent's session history could execute unauthorised tools by forging a tool-confirmation response, because the framework did not verify that the target tool was registered to the executing agent, did not validate whether the tool actually required confirmation, and did not match the confirmation arguments against the original tool-call event in the history. In other words: the human-approval gate could be satisfied without a human. Google, as the assigning CNA, scored it 9.3 Critical on CVSS v4.0; NVD had not published its own assessment at the time of writing. The AWS flaw, CVE-2026-18830, is described by NVD as insufficient input validation letting an authenticated remote user "execute configured tools bypassing model invocation and security controls"; AWS scored it 8.6 High and says its managed fix required no customer action. Google's and Vercel's fixes ship as package updates that self-hosted operators have to apply themselves.

The softer version of the same failure needs no CVE. Any governance rule that lives in the agent's instruction context can be summarised away by the agent's own context management. In February 2026, Meta AI security researcher Summer Yue described on X — in an account TechCrunch reported but explicitly said it could not independently verify — an OpenClaw agent deleting her email in what she called a "speed run" while ignoring stop commands from her phone. She had tested it first on a smaller "toy" inbox, where it behaved. Yue's own hypothesis, as reported, was that the volume of data in her real inbox triggered compaction, and that the agent skipped her last instruction and reverted to its earlier behaviour. Asked whether she had been deliberately testing guardrails, she said she had not.

Whatever the precise mechanism there, the point practitioners drew from it at the time holds: a prompt is a request, not a control. Testing for this means running full-length operational scenarios rather than happy-path cases — fill the context, introduce the latency and API errors production actually produces, and check whether the escalation logic still fires after forty turns and a compaction event. For gates enforced outside the model, test the gate's own logic: can a confirmation be satisfied for a tool that never required one, or with arguments that don't match the call it supposedly approves?

3. Multi-step chaining that bypasses per-action policies

This is the subtlest failure mode and the hardest to catch. A policy checks each action individually. Can this agent search the web? Approved. Can it read a public repository? Approved. Can it authenticate to a system with credentials it holds? Approved.

What the per-action check does not catch is the composition. Run those three in sequence and you have the Gemini exercise: search on a company name, find credentials in a public repository, use them. No single step is obviously out of bounds. The chain is the breach.

Each individual permission was correctly granted; the aggregate behaviour exceeded the intended scope. That is the scope violation problem at sequence level, and it gets worse with delegation, because a subagent tree multiplies the number of permitted sequences without multiplying anyone's ability to enumerate them. Human reviewers are not a reliable backstop here either — we looked at how approval scope gaps survive review in AI Agent Permissions: Humans Miss 1 in 3 Threats.

Testing for it means thinking in sequences rather than steps: which combinations of approved actions, executed in order, produce an outcome your policies were written to prevent?

How to Test the Governance Layer

Treat the governance plane as a system under test, with the rigour you would apply to any other infrastructure component.

Test boundary conditions, not just happy paths. If a cost limit fires at $10, don't only verify that a $5 run completes. Run something that hits $10.00, $10.01 and $50. Verify the limit fires at the right threshold, stops cleanly, and doesn't leave partial state the agent can resume from. Confirm too that the thing your ceiling counts is the thing your bill counts, and that its scope survives fan-out into subagents and retries.

Test governance under load and context pressure. Run the longest scenarios you expect in production, with realistic tool-call volumes and latencies. Does the escalation policy survive a forty-turn conversation? Does it survive a compaction event?

Test the authorisation path, not only the model's judgement. CoreBreak's lesson is that threat models stopping at could the model be persuaded to call an unsafe tool leave the more direct question unasked: can the dispatch layer be reached without the model at all? CSA's recommendation is to verify that the code executing a tool call checks that the call originated from a model turn belonging to the correct agent and session, with arguments matching what the model returned — rather than merely checking that the payload is well-formed. Where a confirmation gate protects a sensitive tool, check that the confirmation logic validates tool ownership, confirmation requirement and argument integrity against the original recorded call.

Test adversarial inputs against the governance layer specifically. Not whether the agent produces good outputs — whether the policy boundaries hold. Can the agent be prompted into requesting a permission it shouldn't have? Can it be steered into chaining permitted actions into a prohibited outcome? If you rely on output checks, test whether they hold when an output is crafted to pass the check while violating its intent — the gap between a passing check and a correct result is its own failure class, which we covered in 200 OK Isn't Success.

Don't let an aggregate score stand in for a governance result. A suite that reports a single pass rate can hide a specific control that never fired; that masking effect is exactly what we found in One Score Masks a 91-Point Drop, and in orchestration testing where every task passes and the risk is still there.

Use execution replay to investigate failures. When a control doesn't fire, you need to see what the agent saw at the moment it should have. Replaying the execution record — the tool calls, the context states, the policy evaluations, and the governance state active at the time — is how you diagnose a governance failure without reconstructing it from logs afterwards.

Treat policy changes as code changes. Tightening a cost limit, adding a tool restriction, changing an escalation trigger: run a regression against the governance suite. The hard part is rarely the individual change. It is catching how one policy modification interacts with the rest of the stack.

How Waxell handles this: Waxell's Policies are defined in the governance plane and applied by reference, so agents don't carry local copies of policy logic — a rule held in the plane cannot be summarised out of an agent's context, because it was never in that context to begin with. Waxell Observe auto-instruments 160+ Python libraries in two lines (after pip install waxell-observe: import waxell_observe as waxell, then waxell.init() before your other imports) and enforces 50+ policy categories during execution, so the same rules your agents run under are the ones you can write tests against. Those tests run in Waxell's pre-production Testing environment: the same policies, budgets and orchestration paths as production, in a sandbox that cannot mutate production state, and deliberately built to exercise boundary conditions and failure paths. Every run leaves a durable execution record of which policies were evaluated and what they decided. For workflows where the stakes justify building inside the governed environment rather than instrumenting into it, Waxell Runtime gates each step before it executes.

Waxell is free to start — the Free plan covers 10,000 traced executions a month, and managed Runtime is included on Business. See Waxell govern your agents, live — book a 30-minute demo →

Behavioral testing is necessary. It is not sufficient. Teams running serious production agent systems are arriving at two distinct QA disciplines: one for the agent, one for the control layer above it. The second is still the thinner of the two — and Gartner's projection that 40% of enterprises will demote or decommission autonomous agents by 2027 over governance gaps found after the fact is a forecast about what happens when it stays that way. The most instructive failure of 2026 so far happened inside a testing environment, to a control nobody had thought to test. That is the gap to close.

Frequently Asked Questions

What is the difference between behavioral testing and governance testing for AI agents? Behavioral testing checks whether an agent completes its intended tasks — the right tools, in the right order, with the right outputs. Governance testing checks whether the control layer above the agent — cost limits, escalation policies, content filters, permission boundaries — correctly intercepts and stops out-of-policy behaviour when it occurs. Both are necessary. The evidence suggests most teams have invested far more in the first.

Can governance controls fail even if the agent is behaving correctly? Yes. Governance controls fail independently of agent behaviour. Common failure modes: cost limits with the wrong enforcement trigger or the wrong unit of account, approval gates whose confirmation logic can be satisfied without a human, escalation policies embedded in agent context that get dropped under context pressure, and per-action permission checks that miss multi-step sequences violating policy intent in aggregate. In each case the agent may be doing exactly what it was asked to do.

How do you test whether a human-approval gate can be bypassed? Test the gate's logic, not just the happy path through it. Confirm that the component processing an approval verifies the tool belongs to the agent executing it, that the tool actually required confirmation, and that the confirmation's arguments match the original recorded call. CVE-2026-18236 in Google's Agent Development Kit is the worked example of what happens when it doesn't: per NVD, a forged confirmation event injected into session history could execute unauthorised tools, because none of those three checks were performed.

How do you test whether an escalation policy will actually fire in production? Run scenarios designed to trigger the escalation condition, not normal runs that happen to stay in bounds. Then test the boundary: at the exact threshold, one step past it, and under high context pressure or API latency. If you rely on escalation instructions inside the agent's context window, test whether they survive a long conversation and a compaction event.

What is multi-step chaining in the context of agent governance? Multi-step chaining is when an agent composes individually permitted actions into a sequence that violates the intent of your policies, without any single action tripping a check. It is hard to detect because per-action checks are the norm, and it gets harder with delegation, where a subagent tree multiplies the number of permitted sequences. Testing for it means asking which combinations of approved actions, run in order, produce an outcome your policies were designed to prevent.

Why is governance testing separate from security testing? They overlap, but they are not the same exercise. Security testing asks whether an adversary can break the system. Governance testing asks whether the operational control layer — cost limits, approval gates, tool restrictions — works correctly under normal and edge-case conditions. Both answers need to be yes, and the CoreBreak disclosures are a reminder that the boundary between them runs through the same code.

What percentage of deployed AI agents have documented guardrails? According to the 2025 AI Agent Index, which documented 30 deployed AI agents and is published at ACM FAccT 2026, 9 of 30 agents had no guardrails documented at all, and sandboxing or VM isolation was documented for only 9 of 30. Among the 13 enterprise agents in the sample, 7 described options for setting up guardrails but no sandboxing or containment. Across the safety, evaluation and impact fields tracked, 135 of 240 had no public information available.

What does Gartner predict about AI agent governance failures? In a 26 May 2026 press release, Gartner predicted that by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps identified only after production incidents occur. Its recommendation is proportional governance: classify agents by autonomy level rather than treating governance as binary. Gartner's Varma notes that at the approval tier, without strong security testing and clear approval workflows, approvals can degrade under time pressure, "creating a false sense of safety while expanding the attack surface."

Sources

FAQ

What is the difference between behavioral testing and governance testing for AI agents?

Behavioral testing checks whether an agent completes its intended tasks — the right tools, in the right order, with the right outputs. Governance testing checks whether the control layer above the agent — cost limits, escalation policies, content filters, permission boundaries — correctly intercepts and stops out-of-policy behaviour when it occurs. Both are necessary. The evidence suggests most teams have invested far more in the first.

How do you test whether a human-approval gate can be bypassed?

Test the gate's logic, not just the happy path through it. Confirm that the component processing an approval verifies the tool belongs to the agent executing it, that the tool actually required confirmation, and that the confirmation's arguments match the original recorded call. CVE-2026-18236 in Google's Agent Development Kit is the worked example of what happens when it doesn't: per NVD, a forged confirmation event injected into session history could execute unauthorised tools, because none of those three checks were performed.

How do you test whether an escalation policy will actually fire in production?

Run scenarios designed to trigger the escalation condition, not normal runs that happen to stay in bounds. Then test the boundary: at the exact threshold, one step past it, and under high context pressure or API latency. If you rely on escalation instructions inside the agent's context window, test whether they survive a long conversation and a compaction event.

What percentage of deployed AI agents have documented guardrails?

According to the 2025 AI Agent Index, which documented 30 deployed AI agents and is published at ACM FAccT 2026, 9 of 30 agents had no guardrails documented at all, and sandboxing or VM isolation was documented for only 9 of 30. Among the 13 enterprise agents in the sample, 7 described options for setting up guardrails but no sandboxing or containment. Across the safety, evaluation and impact fields tracked, 135 of 240 had no public information available.

What does Gartner predict about AI agent governance failures?

In a 26 May 2026 press release, Gartner predicted that by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps identified only after production incidents occur. Its recommendation is proportional governance: classify agents by autonomy level rather than treating governance as binary. Gartner's Varma notes that at the approval tier, without strong security testing and clear approval workflows, approvals can degrade under time pressure, "creating a false sense of safety while expanding the attack surface."