Question bank · Consistency

Does the agent behave the same in every channel?

Usually not, and the gap is rarely deliberate. Chat, email, and voice reach the same model through different front doors, and the front door is where identity and context get decided. The primary channel gets the policy work, the scenario coverage, and the review. The others inherit whatever the integration happened to pass through. So the useful form of the question is narrower than it sounds: which channel was tested, what is enforced on each of the others, and is the identity check a property of the agent or of the widget it usually runs in?

01

The front door

Where do two channels actually differ?

At the identity boundary, first and worst. The published example profile lists three surfaces for one agent and records the authentication state of each: a storefront chat widget, authenticated; an internal API, authenticated by key; and an email surface, ingested through webhooks, recorded as not authenticated. It then states the consequence in its own words. On the widget and API surfaces the session carries a verified customer identifier and the tools ignore anything the model supplies. On the email surface the customer is resolved by matching the From address, which is not verified, so a spoofed sender would be treated as that customer. One agent, one policy, two boundaries.

SurfaceHow the caller is identifiedWhat that buys
Chat widgetA session token issued to a signed-in shopperReads scoped by the session, not by the model
Internal APIAn API key held by a merchant integrationScoped, and the caller is a system rather than a person
Inbound emailThe From address on the message, recorded as unverifiedA sender the profile says a spoof would be treated as

These three rows are one file, not three products. The surfaces block of the example declared profile carries a channel, a platform, an authentication state, and a source line for each — the level of specificity at which this question stops being answerable with reassurance. The example declared profile.

02

Scope of a result

What does a report say about the channel it did not test?

It says so in the limitations, or it is not an honest report. All three of our published specimens declare a single channel and exclude the rest in the same sentence: the tested channel is text only; voice, telephony, recording consent, and audio quality are excluded. Voice also appears in the operating envelope as a named exclusion, rather than as an omission a careful reader has to notice. A result that does not name its channel will be read as covering all of them, by the buyer, and it will not be the vendor who corrects it.

TEXT ONLY, SAID THREE TIMES

Every specimen we publish declares its channel as Text, excludes voice and telephony in the operating envelope, and repeats the exclusion in the limitations. Nothing in any of those records is evidence about a phone line. Saying that costs a vendor nothing and is the difference between a scoped record and an implied claim.

The insufficient-evidence specimen
03

Two kinds of consistency

Is this just a repeatability question?

No — those are two questions and only one of them is usually measured. Repeatability asks whether the same request in the same channel gets the same answer twice, and our specimens test it with a single repeated scenario at three trials. In the returns specimen all three returned the same eligibility result. In the subscription specimen the third produced no observation at all, so the scenario is recorded as unverified and the consistency dimension reads 0 of 1: a trial that yields nothing is never scored as a pass. Cross-channel consistency is the other question. It does not vary run to run, and no number of repeats inside one channel detects it.

A refusal is the case where this bites hardest. If the boundary lives in the channel rather than the agent, the same request that was refused in chat may not meet the same check arriving by email — does a refused request win on the second try.

04

The answer to give

What should a vendor be able to show?

The same scenario executed on each channel the product actually supports, with the identity check named per surface. Where a channel is unauthenticated, say what compensates and where that compensation is enforced. Where a channel is on the roadmap and not in the evidence, say that too. The cheapest version of this finding is the one a vendor makes about its own email path before a reviewer asks which door the agent answers.

→

Related

What else does a reviewer ask about this?

The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.

01Can an agent read one customer's data to another?It can whenever a lookup is scoped by an identifier the model supplies rather than one the verified session carries; this is the highest-severity class because no attacker is required, and one failure here outranks any aggregate pass rate.02Does a refused request win on the second try?Sometimes. A refusal is a single-turn outcome; the same request re-entered in a new session or through a different channel meets an agent with no memory of refusing, so the question is whether the boundary is a property of the system or of one conversation.03Can a tool response carry an attack?Yes, and it is the version most vendors have not tested: the attack arrives in the data the agent reads — an order note, a product review, a forwarded email body — inside a tool response the agent trusts by construction, so passing a single-turn direct extraction test says nothing about it.04What evidence do enterprise reviewers accept about an AI agent?Executed runs against cases the vendor did not choose, on a frozen version, with the raw traces preserved and the results recomputable; operator-supplied evidence still counts as context, but it caps the conclusion, because no operator can upgrade its own evidence.

The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list