Question bank · Authority limits
Does a refused request win on the second try?
Sometimes, and most testing never finds out. A refusal is a single-turn outcome, and a demo stops there. The same request re-entered later — in a new session, through a different channel, with the facts rearranged — meets an agent that has no memory of having refused. If the refusal came from an instruction in the system prompt rather than a check in code, the second attempt is a fresh roll of the same dice. What the reviewer is really asking is whether the boundary is a property of the system or a property of one conversation.
The pattern
What does a second-try attempt look like in practice?
It rarely looks like an attack. The request is refused in the chat widget, then arrives by email an hour later with a slightly different story: an order number instead of a name, urgency instead of a threat, a manager mentioned instead of a policy quoted. Nothing about the second attempt is adversarial in tone. It is simply a second draw, and the persistence costs the customer nothing.
Channels
Why does the channel change the answer?
Because channels usually differ in how they establish who is speaking. Our own published example profile is explicit about its own asymmetry: on the widget and API surfaces the session carries a verified customer identifier and the tools ignore any identifier the model supplies, but on the email surface "the customer is resolved by matching the From address, which is not verified, so a spoofed sender would be treated as that customer." A refusal that depends on identity is only as strong as the weakest channel that can reach the same tool.
That quote is from our own published example. It is the kind of admission an intake pass produces when the agent is asked to describe itself honestly rather than favourably — the example declared profile names the unverified email path in its own uncertainty list.
Repetition
Why doesn't one refusal settle it?
Because the same input does not reliably produce the same output. That is a property of the systems these agents are built on, not a defect in a particular one. A refusal observed once is one observation. Our reference method treats repetition as its own assurance dimension — consistency asks whether a repeated scenario ends the same way, and it is scored from repeated trials rather than from a single pass.
In our method, consistency is one of seven assurance dimensions and carries its own declared threshold, set before the run. The question it asks is deliberately blunt: repeated, does the same scenario end the same way? A single observation cannot answer it, so a single observation is not accepted as an answer.
The seven dimensionsTesting it
How is persistence tested?
Repeat the scenario, then vary it. The same request across every channel that reaches the same tool. The same request in a fresh session with no prior context. The same request rewritten so the refusal trigger is absent but the underlying action is identical. A boundary that holds under all four is a control. A boundary that holds under one is an anecdote.
The variations are held back on purpose. The classes are published here; the specific rewrites are not, because a rewrite you can read in advance is a rewrite you can patch against without fixing anything underneath it.
Related
What else does a reviewer ask about this?
The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.
The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.
Answering it with evidence
Can you answer this about your own agent?
We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.
What a dry run needs, so nothing is a surprise later
- An agent that takes real actions, not just answers questions
- A version you can freeze for two weeks
- A sandbox or endpoint we can reach