Question bank · Definitions
What is a declared profile?
A declared profile is an agent's self-declared capability inventory: the tools it can call, what it is allowed to write, the actions that move money, the customer data it can reach, when it escalates, and the version identifiers that say which build the declaration describes. It is produced by the vendor's own tooling, reading the vendor's own repository. It is a declaration, not evidence — nothing in it has been independently observed. Its value is that it makes the vendor's claim specific enough to test.
Contents
What goes in a declared profile?
Subject and version digests. Purpose, intended users, and environment. Autonomy — the oversight model, the financial authority and the basis on which it is enforced, and what the agent can and cannot access. Capability claims, each with a success definition and a consequence if it is wrong. Then the inventory below, and a list of what the declaration could not determine.
One shape, whatever wrote it. The same schema comes out of every harness, so two agents can be compared field by field. A complete worked example — synthetic, and it says so in its own metadata.
Declaration, not evidence
How is a declared profile different from evidence?
Completely, and the file says so about itself. The declaration field in our published example reads: self-declared by the vendor's own tooling from the vendor's own repository; a declaration of intended capability, not evidence of behavior; nothing in it has been independently observed. A declared profile tells you what to test. It cannot tell you what happened, because nothing has run.
A declaration that claimed to be evidence would be the first thing worth distrusting, so the schema carries the disclaimer as a field rather than as a footnote. What a reviewer does with a declared profile is decide what to execute against it.
What counts as evidenceThe uncertainty list
Which part of a declared profile is most useful?
The uncertainty list. A profile that admits "cancel_order sends no idempotency key — whether the orders service deduplicates retries was not determined from this repository" has just named the first thing worth testing. So has one that admits an email surface resolves a customer from an unverified sender address. An agent describing itself will miss things: the tool it forgot, the path it never mentions, the action it does not think of as spending money. Finding what it did not mention is the job that comes after.
An empty uncertainty list is a finding. No intake pass over a real repository determines everything. A profile that claims to have resolved every question has either been edited for presentation or has not looked hard enough, and either way it is the wrong file to hand a reviewer.
Producing one
How do you produce a declared profile?
Run the intake skill for your harness against your own repository. It is free, open source, runs on your machine, and sends nothing to us; the output is one standard JSON file in the same shape for every harness. Ten minutes. Then read the uncertainty list before anyone else does.
Related
What else does a reviewer ask about this?
The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.
The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.
Answering it with evidence
Can you answer this about your own agent?
We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.
What a dry run needs, so nothing is a surprise later
- An agent that takes real actions, not just answers questions
- A version you can freeze for two weeks
- A sandbox or endpoint we can reach