Question bank · Evidence standards

What evidence do enterprise reviewers accept about an AI agent?

Executed evidence about cases the vendor did not choose, on a frozen version, with the traces preserved. Everything else is context. A policy says what should happen. An eval suite says what the vendor thought to test. A demo says what happened once, on an example the vendor picked. Only a run against independently selected scenarios, with the raw observations kept and the results recomputable from them, answers the question a reviewer is actually asking — which is what the agent does when nobody is choosing the example.

01

Grades

Are all forms of evidence equal?

No. Our method distinguishes exactly one boundary that matters: whether the evidence was captured at an observation point the operator controls. G0 and G1 are boundary-attested. G2 and G3 are operator-supplied — captured by, or handed over by, the party being assessed. Operator-supplied evidence is still worth having, and it is still read, but it caps the conclusion at "limited" no matter how well the run went.

RankFormWhat it can support
01A policy or system cardStates intent. Answers no behavioral question.
02An internal eval suiteCovers what the team thought to test. Selected by the party being assessed.
03A demo or recorded sessionOne run, one case, chosen by the vendor. Not a sample of anything.
04An executed run on independently selected casesAnswers the behavioral question — and is capped at limited while the operator supplies the evidence.
02

The cap

Why can't a vendor upgrade its own evidence?

Because if it could, the grade would measure nothing. The cap is applied in code, in a fixed precedence order, before any aggregate is considered. In the current version the schema will not even accept an operator-declared G0 or G1: a manifest that claims one fails validation, with the message that boundary attestation must be assigned by a verifier that does not exist yet. That means our own published sample carries a G3 grade and cannot conclude "supported." The honest answer to "can you certify this agent" is still no.

THE CEILING IS LIMITED

G2/G3 evidence caps an otherwise passing result at limited. That rule sits fourth in a five-rung precedence ladder, and the rungs above it — a verified high or critical failure, and missing or non-executable evidence — are worse. Our own sample runs into the cap. We publish the rule rather than the exception, because a standard that exempts its author is not a standard.

The precedence ladder
03

Traces

What does a preserved trace mean?

That every finding points back to the observation it came from, and that the whole file recomputes from raw evidence rather than from a summary. In the sample, the failing check is reproduced with the field it read, the comparison it applied, the expected value, and the observed value, so a reader can disagree with the conclusion by inspecting the same material rather than by taking a different view of a paragraph.

A check that cannot execute does not quietly pass. It produces insufficient evidence, which is its own conclusion. Failing closed is the difference between a report that describes a run and a report that describes a hope. See it in the sample.

04

Integrity is not authenticity

What does a hash chain actually prove?

Detect mutation, and nothing more. A hash chain proves the record you are holding is the record that was produced. It does not prove who observed the agent, or that anything was observed at all — anyone who can rewrite all the evidence can recompute both hashes. We label that gap rather than describing a content hash as an attestation, because it is the easiest thing in this field to oversell.

Ask any vendor who shows you a hash. Integrity means the file was not altered after it was written. Authenticity means someone independent saw the thing happen. They are different pieces of engineering and only one of them is easy.

→

Related

What else does a reviewer ask about this?

The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.

01Does SOC 2 cover AI agent behavior?No. SOC 2 audits organizational controls — access, encryption, change management, incident response — over a period of time, and never executes the agent, so a clean report is compatible with an agent that refunds twice or answers one customer with another customer's data.02What does it mean to freeze an agent version?Recording a digest of the four things that determine behavior — model, prompt, policy, and toolset — so any later change is detectable; a git commit is not enough, because two deploys of the same commit are different agents if the model alias resolved to a new checkpoint.03What is a declared profile?An agent's self-declared capability inventory — tools, write authority, money-touching actions, data access, escalation, and version identifiers — produced by the vendor's own tooling from the vendor's own repository; it is a declaration, not evidence, and its value is making the claim specific enough to test.04How fast does a human actually take over?Escalation is a claim about latency and completeness, not existence: how long before a person is looking, what they receive with the conversation, and whether a retry or a queued tool call keeps acting in the gap — and the strongest answer names a code-enforced stop rather than an instruction to stop.

The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list