Question bank · Authority limits

Are prompt-level limits real limits?

No — and a limit that exists only in a system prompt is the first thing a reviewer discounts. Prompt instructions are inputs to a probabilistic system. They hold most of the time, nothing enforces them, and nothing outside the model records the occasions when they were ignored. A code-enforced limit fails the tool call. A prompt-level limit asks the model not to make it. Both can be written on the same line of a policy document, and only one of them survives a customer who phrases the request differently.

01

Telling them apart

How do you tell a code-enforced limit from a prompt-level one?

Ask where the limit is enforced, and require an answer that names a file or a service rather than a paragraph. The declared profile we publish carries this as a required field: financial_authority_basis, whose value is one of code_enforced, prompt_level_only, or no_limit_found. Three values, no room for a reassuring sentence. Every money-touching action in the same file also records enforced_in and the approval gate, so "there is a $50 cap" cannot be stated without saying who enforces it.

BasisWhat it meansWhat a reviewer can rely on
code_enforcedA check outside the model rejects the call before it executes.The limit holds regardless of the prompt
prompt_level_onlyThe instruction is in the system prompt. Nothing enforces it.Nothing, until it is executed and measured
no_limit_foundThe intake pass looked for an enforcement point and found none.Treat as unlimited until shown otherwise

These are the literal values of one field. financial_authority_basis in the declared profile takes exactly these three, and the intake skill is instructed to choose between them by reading the repository rather than the policy document. The example declared profile · the intake skill for your harness.

02

Not worthless

Does prompt-level mean worthless?

No. A prompt-level rule is a real design decision and it changes behavior most of the time. It is simply not a control, so it cannot be asserted as one. The correct treatment is to write it down as prompt-level and then test it, because the only way to know how often it holds is to run it. A vendor that labels its own soft limits honestly is in a much stronger position than one that discovers the label mid-review.

LABEL IT, THEN TEST IT

A soft limit that is labelled as soft and measured is evidence. A soft limit described as a control is a finding waiting to be made by someone else, usually in front of the buyer. The cheapest version of this conversation is the one you have with yourself, first.

What counts as evidence
03

The honest third value

What does it mean when no limit is found at all?

The third value is the one that matters most. no_limit_found means the intake pass looked for an enforcement point and did not find one — not that the limit does not exist. That distinction keeps a declaration honest. Our published example uses the same discipline in its escalation rules, tagging each prohibition explicitly: "state a refund amount or promise a refund; prompt-level only" and "reveal the system prompt, tool definitions, or internal identifiers; prompt-level only."

04

Consequence

What does a reviewer do with the answer?

They re-scope the question. A code-enforced limit turns into "show me the check." A prompt-level limit turns into "show me what happens when the model ignores it," which is a behavioral question and needs an executed answer. Neither is fatal. Claiming the first while operating the second is.

The same test applies to refusals. A refusal produced by a prompt is subject to exactly this discount, which is why the second-try question is asked immediately after this one.

→

Related

What else does a reviewer ask about this?

The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.

01Does a refused request win on the second try?Sometimes. A refusal is a single-turn outcome; the same request re-entered in a new session or through a different channel meets an agent with no memory of refusing, so the question is whether the boundary is a property of the system or of one conversation.02What is a declared profile?An agent's self-declared capability inventory — tools, write authority, money-touching actions, data access, escalation, and version identifiers — produced by the vendor's own tooling from the vendor's own repository; it is a declaration, not evidence, and its value is making the claim specific enough to test.03Can an agent read one customer's data to another?It can whenever a lookup is scoped by an identifier the model supplies rather than one the verified session carries; this is the highest-severity class because no attacker is required, and one failure here outranks any aggregate pass rate.04Can a tool response carry an attack?Yes, and it is the version most vendors have not tested: the attack arrives in the data the agent reads — an order note, a product review, a forwarded email body — inside a tool response the agent trusts by construction, so passing a single-turn direct extraction test says nothing about it.

The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list