Question bank · Version binding

What happens when a model swap changes behavior?

Someone notices, and which someone is a design decision rather than luck. The provider updates the checkpoint behind an alias. Nothing shipped on the vendor's side, no config moved, no version number changed. The vendor's regression suite may or may not run again, and if it does it runs the cases that were chosen while the previous checkpoint was in production. The customer runs the case nobody wrote a test for. Whether the vendor gets there first turns on one thing: whether the deployed version records what the alias resolved to, or only the alias.

01

The alias

Why is a model alias not a version?

Because an alias is a name for whatever sits behind it today. The published example declared profile is unusually honest about this. Its model identifier is recorded as a vendor-hosted alias, annotated that the underlying model is set outside the repository, and the file then says the quiet part in its own uncertainties: the model digest is the hash of the alias string, not of a model. A version record built on that cannot detect a swap, because nothing it hashes changes when the swap happens.

THE DIGEST OF A NAME

The example profile records four digests — model, prompt, policy, toolset — and then discloses that one of them is not what it appears to be: the model digest is the hash of the alias string, not of a model. Writing that down is worth more than the digest was. It tells a reviewer exactly which of the four bindings will not hold.

The example declared profile
02

What a suite has to be

What would a regression suite have to be to catch it?

It has to re-run on the change rather than on the calendar, and it has to score more than text. Our specimens execute between 25 and 46 deterministic checks each, and the interesting ones are not the response assertions: 5 final-state and 7 trajectory checks in the thinnest of the three, 15 and 11 in the fullest. An agent that starts taking a different route to the same sentence passes every output comparison and fails a trajectory check. A suite made only of output comparisons is watching the part of the behavior that moves last.

One repeated scenario is not a distribution. Each of our specimens repeats exactly one scenario, three times, which is enough to notice gross nondeterminism and not enough to characterise a shift. In the subscription specimen even that was not achieved: the third trial produced no observation, so the scenario is recorded as unverified rather than as two passes. The insufficient-evidence specimen.

03

Order of discovery

Who notices first?

There are three possible orders and only the first two are chosen. A recorded digest can catch it before anything is judged by it. A suite can catch it if the change is what triggers the suite. Otherwise it is the customer, who is the cheapest detector to build and the most expensive one to use.

01The version record noticesA recorded digest mismatches on the next run, before any decision rests on the new behavior. Unavailable if the digest is a hash of an alias.
02The regression suite noticesOnly if the change is the trigger. A suite on a monthly schedule leaves up to a month of exposure between the swap and the signal.
03The customer noticesThe default. It costs nothing to build and it arrives as an incident with an account name attached to it.
04

The answer to give

What should a vendor be able to show?

The resolved checkpoint captured at run time, not the alias that was requested; the trigger that reruns the suite when that value changes; and a result dated against that specific resolution. If the provider does not expose the resolved checkpoint, write that into the version record the way our example profile does. An admitted gap in a version identifier is a much smaller problem than a version identifier that is quietly a nickname.

A swap is a renewal trigger, and every specimen says so. The trigger printed in each record is any change to the model, prompt, policy, toolset, action authority, or material operating environment. What that means for a result already in a buyer's hands is the neighbouring question: is a passing test from March still valid in April.

→

Related

What else does a reviewer ask about this?

The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.

01What does it mean to freeze an agent version?Recording a digest of the four things that determine behavior — model, prompt, policy, and toolset — so any later change is detectable; a git commit is not enough, because two deploys of the same commit are different agents if the model alias resolved to a new checkpoint.02Is a passing test from March still valid in April?Only if nothing that determines behavior changed, and something usually did: the renewal trigger is any change to the model, prompt, policy, toolset, action authority, or material operating environment — and a provider swapping a checkpoint behind a model alias meets it without a deploy on the vendor's side.03What evidence do enterprise reviewers accept about an AI agent?Executed runs against cases the vendor did not choose, on a frozen version, with the raw traces preserved and the results recomputable; operator-supplied evidence still counts as context, but it caps the conclusion, because no operator can upgrade its own evidence.04What is a declared profile?An agent's self-declared capability inventory — tools, write authority, money-touching actions, data access, escalation, and version identifiers — produced by the vendor's own tooling from the vendor's own repository; it is a declaration, not evidence, and its value is making the claim specific enough to test.

The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list