Open navigation menu
How should human approval be enforced for every AI-agent side effect?
In this synthetic benchmark, only a last-mile exact-action broker produced zero false permits. It bound approval to the canonical payload, identity, tenant, context, expiry, and a one-time nonce before the side effect executed.
72 frozen cases
4 controls
1,440 decisions
Dataset v1.0.1
The exact-action broker was the only control with zero false permits.
| Control | Decision accuracy | False permits | Reapproval recall | Replay blocked |
|---|---|---|---|---|
Workflow booleanworkflow_boolean | 61.1% | 40.0% | 0.0% | 0.0% |
Tool-name gatetool_name_gate | 68.1% | 32.9% | 20.0% | 0.0% |
Scoped-action gatescoped_action_gate | 88.9% | 10.0% | 76.0% | 50.0% |
Exact-action brokerexact_action_broker | 100.0% | 0.0% | 100.0% | 100.0% |
The weaker controls failed on different parts of the approved action.
Payload changed after approval
A tool name was too broad
A reusable approval was replayed
A frozen 72-case dataset was evaluated five times against each control.
- 01The base matrix combines five side-effect categories, four post-approval mutations, and three approval states for 60 cases.
- 02Twelve controls cover exact execution, canonical JSON, denial, timeout, replay, identity, tenant, context, display mismatch, child calls, and alternate paths.
- 03Each case runs independently five times against four controls; the runner verifies the frozen CSV checksums before execution.
- 04A false permit means a side effect executed when the frozen expected decision was deny or reapprove.
Each implementation adds one more binding boundary.
- 01workflow_boolean asks only whether the workflow is marked approved.
- 02tool_name_gate also checks the named tool, but ignores its target and payload.
- 03scoped_action_gate checks tool, operation, target, principal, tenant, and context, but not the canonical payload or one-time use.
- 04exact_action_broker checks the complete displayed action at the last side-effect boundary and consumes a nonce once.
Download the frozen inputs, ground truth, raw decisions, scores, manifest, and runner.
This benchmark isolates approval binding; it is not a security certification.
These are Moonveil reference implementations, not tests of named agent frameworks or vendors.
The benchmark does not cover distributed nonce storage, real concurrency, key rotation, UI deception, durable log integrity, or network faults.
The HMAC key is intentionally public and demonstrates action binding, not production secret management.
Synthetic results are not a security certification and do not establish regulatory compliance or production safety.
Publication record
- Author
- Moonveil AI
- Published and updated
- July 24, 2026
- Independent security review
- Not performed
- Commercial disclosure
- Moonveil AI provides AI Agent implementation and evaluation services.