Question bank · Evidence standards
Does SOC 2 cover AI agent behavior?
No. SOC 2 audits organizational controls — access management, encryption, change management, incident response — over a period of time. It never executes your agent. A SOC 2 report can be entirely clean while the agent inside the product issues the same refund twice, answers one customer using another customer's order, or repeats its own system prompt when asked in the right way. Those are behaviors. No control test observes them, and no auditor is asked to.
Scope
What does SOC 2 actually test?
It tests that the controls a company says it operates were operating. Who could reach production, whether changes were reviewed, whether data was encrypted, whether incidents were handled the way the policy says. Those are properties of the organization. An AI agent that takes actions introduces a second thing that can fail — the agent's own decisions inside the boundary the controls established — and SOC 2 has no opinion about it.
This is not a criticism of SOC 2. It does the job it was designed for, and an agent vendor without one has a different problem. The point is narrower: a report about controls cannot be made to answer a question about conduct, and reviewers know it.
The gap
What does the reviewer ask instead?
Reviewers ask what the agent does once it is already inside the systems SOC 2 covers. The pattern is consistent: for every area the report answers, there is an adjacent question about behavior that it does not.
There is no standard form for the right-hand column. It arrives ad hoc, usually live on a technical call, and “it won't do that” is not an accepted answer.
Behavioral controls
Is there a standard that does test agent behavior?
Partly. AIUC-1, the third-party certification standard for AI agents, splits its requirements between organizational process and observed behavior: 26 of its 51 published requirements test what the agent does rather than what the organization documents — 12 under Safety, 10 under Security, 4 under Reliability. That is the clearest published statement that roughly half of agent assurance is not a paperwork exercise. Producing evidence for those controls still means running the agent.
26 of AIUC-1's 51 requirements — 12 Safety, 10 Security, 4 Reliability — test behavior, not paperwork. Source: the AIUC-1 control list. We are not affiliated with AIUC and we do not certify against AIUC-1; we cite it because it is the clearest public accounting of how much of agent assurance cannot be documented into existence.
How we reach a conclusionWhat closes it
What closes the gap?
Executed evidence against the specific actions your agent can take. Not a policy describing intent, and not an eval suite covering what your team thought to test — a run on a frozen version, against cases the vendor did not choose, with the traces preserved so a reviewer can check the finding rather than trust it.
What that looks like on paper. Our published sample is a synthetic agent that passed 13 of 14 trials and still had its conclusion withheld, because one critical failure in the privacy and security dimension outranks the aggregate. Read the sample — it is the shape of an answer to the right-hand column above.
Related
What else does a reviewer ask about this?
The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.
The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.
Answering it with evidence
Can you answer this about your own agent?
We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.
What a dry run needs, so nothing is a surprise later
- An agent that takes real actions, not just answers questions
- A version you can freeze for two weeks
- A sandbox or endpoint we can reach