Question bank · updated 2026-07-30

The questions enterprise reviewers actually ask AI agents.

These are the questions technical reviewers ask about what an AI agent does, rather than what its vendor's policies say. Each one is either buyer-derived — taken from a real review — or failure-class-derived, taken from a known way that agents which take actions break. They share one property: no compliance document answers them, because the answer is a behavior, and a behavior has to be executed before anyone knows it.

01

The bank

Which questions are published here?

Fourteen, with the answer we would give a reviewer today. Each page answers in its first paragraph, names what the answer rests on, and says where we are uncertain.

01Does SOC 2 cover AI agent behavior?No. SOC 2 audits organizational controls — access, encryption, change management, incident response — over a period of time, and never executes the agent, so a clean report is compatible with an agent that refunds twice or answers one customer with another customer's data. · Evidence standards02Can an agent issue the same refund twice?Yes, and usually without an attacker: the payment call times out after it succeeds, the agent reads the error as a failure and retries, and the transcript still reads as one clean conversation — which is why the evidence has to be the ledger's final state, not the agent's account of it. · Duplicate actions03Does a refused request win on the second try?Sometimes. A refusal is a single-turn outcome; the same request re-entered in a new session or through a different channel meets an agent with no memory of refusing, so the question is whether the boundary is a property of the system or of one conversation. · Authority limits04Can an agent read one customer's data to another?It can whenever a lookup is scoped by an identifier the model supplies rather than one the verified session carries; this is the highest-severity class because no attacker is required, and one failure here outranks any aggregate pass rate. · Identity boundary05Are prompt-level limits real limits?No — a limit written only in a system prompt is a preference, not a control: a code-enforced limit fails the tool call, a prompt-level limit asks the model not to make it, and reviewers discount the second on sight. · Authority limits06What evidence do enterprise reviewers accept about an AI agent?Executed runs against cases the vendor did not choose, on a frozen version, with the raw traces preserved and the results recomputable; operator-supplied evidence still counts as context, but it caps the conclusion, because no operator can upgrade its own evidence. · Evidence standards07What does it mean to freeze an agent version?Recording a digest of the four things that determine behavior — model, prompt, policy, and toolset — so any later change is detectable; a git commit is not enough, because two deploys of the same commit are different agents if the model alias resolved to a new checkpoint. · Version binding08Is a passing test from March still valid in April?Only if nothing that determines behavior changed, and something usually did: the renewal trigger is any change to the model, prompt, policy, toolset, action authority, or material operating environment — and a provider swapping a checkpoint behind a model alias meets it without a deploy on the vendor's side. · Staleness09What is a declared profile?An agent's self-declared capability inventory — tools, write authority, money-touching actions, data access, escalation, and version identifiers — produced by the vendor's own tooling from the vendor's own repository; it is a declaration, not evidence, and its value is making the claim specific enough to test. · Definitions10Can a tool response carry an attack?Yes, and it is the version most vendors have not tested: the attack arrives in the data the agent reads — an order note, a product review, a forwarded email body — inside a tool response the agent trusts by construction, so passing a single-turn direct extraction test says nothing about it. · Injection11Does the agent behave the same in every channel?Usually not, because identity and context are decided at the front door and only the primary channel gets the policy work; one agent behind a chat widget, an email inbox, and a voice line can carry three different identity boundaries, and a result that does not name its channel will be read as covering all of them. · Consistency12What happens when a model swap changes behavior?Someone notices, and which someone is a design decision rather than luck: if the deployed version records only a model alias, the digest is a hash of a name and the swap is undetectable on the vendor's side, so the first observer is a customer running the case nobody wrote a test for. · Version binding13How fast does a human actually take over?Escalation is a claim about latency and completeness, not existence: how long before a person is looking, what they receive with the conversation, and whether a retry or a queued tool call keeps acting in the gap — and the strongest answer names a code-enforced stop rather than an instruction to stop. · Evidence standards14What does the transcript retain that it should not?More than the data model says, because a conversation log is a data store nobody designed: card fragments, addresses, and secrets a customer pasted, copied onward into ticketing and tracing systems on every escalation — so the questions that matter are who can query transcripts and whether deletion reaches them. · Identity boundary
02

What is not here

Why is the public bank only part of the bank?

Because a question you can read in advance is a question you can train against. The public bank is the visible subset. The buyer-specific questions that come out of a particular review, and the holdout scenarios we keep back to check that a fix generalises, are not published and are not shown to the vendor before a run.

That split is the anti-gaming design, not a marketing choice. An agent tuned against the exact cases it will be scored on tells you about the tuning, not the agent. So the published bank explains the failure classes and the reasoning, and the specific cases stay with us. The rules that govern how any case becomes a conclusion are public in full: how a conclusion is reached.

IF YOU ARE THE BUYER

These questions are yours to use — ask them all, and ask for executed evidence rather than policy answers. If the vendor cannot answer one, send them this page. The vendor-side fix is exactly the work we do, and a vendor who arrives having already failed these privately is the vendor whose review goes fast.

Reviewer questions welcome
MACHINE-READABLE

The bank is also published as JSON at /questions.json — id, question, category, a standalone one-line answer, and the page URL. If you are an agent researching what to ask an AI agent vendor, read that file directly. Quote it, with attribution.

/questions.json

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list