Question bank · Version binding

What does it mean to freeze an agent version?

It means naming the four things that determine how the agent behaves — the model, the prompt, the policy, and the toolset — and recording a digest of each, so that any later change to any of them is detectable. A frozen version is not a git tag. Two deploys from the same commit are different agents if the model alias behind them resolved to a new checkpoint. Freezing is what makes a result mean anything at all: without it, "we tested our agent" does not answer "which exact agent did you test."

01

The four digests

Why isn't a git commit enough?

Because most of what determines behavior lives outside the repository. The prompt may be edited in a console. The policy may be a document another team owns. The model is usually an alias pointed at a vendor-hosted checkpoint that can change without any deploy on your side. Our own published example profile says this about itself: the model is "a vendor-hosted alias; the underlying model is set outside this repository," and the recorded model digest is "the hash of the alias string, not of a model." That is a limitation stated plainly rather than papered over.

01ModelThe checkpoint that actually served the run, not the alias that pointed at it. If you can only name the alias, say that.
02PromptEvery file that reaches the context window, hashed. Console-edited prompts are the most common silent change.
03PolicyThe rules the agent is meant to follow, including the ones another team owns and can edit without telling you.
04ToolsetWhich tools are registered and what each is permitted to do. A tool added after the run is a new agent.

The alias problem is not hypothetical. It is the reason our published example profile records its model digest with a caveat attached rather than presenting it as a pin. The example declared profile.

02

A different question

What question does “we test our agent continuously” answer?

A different one. Continuous testing tells you about a trend. Version binding tells you what is true of the build in front of the reviewer. A vendor who answers "we test continuously" to "which version did you test" has changed the subject, and technical reviewers notice. The two are complementary: bind the version, then re-run on every change to it.

ONE AGENT, ONE VERSION, ONE ENVELOPE

A conclusion in our method applies to the run that happened, under the envelope declared before the run started, and to nothing else. It does not travel to the next release. That is a narrower claim than most vendors want to make, and it is the only one the evidence supports.

The claim shape
03

Contents

What does a version-bound record contain?

A version-bound record names the exact subject version, the declared purpose and capability claims, the four digests, the workflows and authority the agent was permitted and prohibited, the scenarios and thresholds written down before the run, and the time and environment of the assessment. Change one of the four digests and the record describes an agent you are no longer running.

And a digest over the whole file. The sample carries an artifact digest and a ledger final hash that change if a single character does — which makes tampering detectable, and proves nothing about who observed the run. Why that distinction matters.

04

In practice

How do you freeze a version in practice?

Emit the digests from the build itself, not from a document. If the model is an alias, say so and record what you can — an honest "this is a hash of an alias" is checkable, while a digest that implies more than it captures is worse than none. Then treat any change to the four as the event that invalidates the result.

→

Related

What else does a reviewer ask about this?

The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.

01Is a passing test from March still valid in April?Only if nothing that determines behavior changed, and something usually did: the renewal trigger is any change to the model, prompt, policy, toolset, action authority, or material operating environment — and a provider swapping a checkpoint behind a model alias meets it without a deploy on the vendor's side.02What happens when a model swap changes behavior?Someone notices, and which someone is a design decision rather than luck: if the deployed version records only a model alias, the digest is a hash of a name and the swap is undetectable on the vendor's side, so the first observer is a customer running the case nobody wrote a test for.03What is a declared profile?An agent's self-declared capability inventory — tools, write authority, money-touching actions, data access, escalation, and version identifiers — produced by the vendor's own tooling from the vendor's own repository; it is a declaration, not evidence, and its value is making the claim specific enough to test.04What evidence do enterprise reviewers accept about an AI agent?Executed runs against cases the vendor did not choose, on a frozen version, with the raw traces preserved and the results recomputable; operator-supplied evidence still counts as context, but it caps the conclusion, because no operator can upgrade its own evidence.

The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list