Question bank · Staleness

Is a passing test from March still valid in April?

Only if nothing that determines behavior changed, and something almost always did. A result describes a run that happened on a specific build under a specific envelope. It is not a statement about the agent in general and it does not roll forward. The renewal trigger recorded in our own sample is explicit: any change to the model, prompt, policy, toolset, action authority, or material operating environment. A provider swapping the checkpoint behind a model alias meets that trigger, and it can happen without a deploy on your side.

01

The trigger

What kinds of change invalidate a result?

Six classes, and only two of them look like a release. A new model checkpoint. An edited prompt. A policy revision made by another team. A tool added, removed, or given a wider scope. A change to what the agent is allowed to do. A change to the environment it operates in — a new channel, a new data source, a new upstream service. Any one of them can move behavior without moving a line of the agent's own code.

01ModelA new checkpoint behind the same alias. No deploy required.
02PromptAny edit to any file that reaches the context window.
03PolicyA rule revised by whoever owns the rules, on their schedule.
04ToolsetA tool added, removed, renamed, or widened.
05Action authorityA limit raised, or an approval gate removed.
06Operating environmentA new channel, a new data source, a new upstream dependency.
THE RENEWAL TRIGGER, VERBATIM

“Any change to the model, prompt, policy, toolset, action authority, or material operating environment.” That line is printed in the sample Factfile itself, next to the artifact digest, so a reader knows what would end the record's validity without asking us.

Read the sample report
02

Silent drift

Why doesn't a model swap announce itself?

Because the interface does not change. The alias resolves, the calls succeed, the latency is similar, and the outputs are still plausible. What changed is the distribution of what the model does in the cases nobody looks at — which is exactly the population a behavioral test was sampling. The regression that matters is the one that only shows up under the failure you injected in March and stopped injecting in April.

This is the practical case for version binding. If the four digests are recorded, a swap is detectable the moment it happens. If they are not, the first signal is a behavior report from a customer. What freezing a version means.

03

Shelf life

How long is a result good for?

As long as the record says, and no longer. The sample we publish is issued on one date and carries an explicit valid-until date roughly three months later, alongside its renewal trigger. The expiry is a ceiling, not a guarantee: the trigger can fire the day after issue, and when it does, the remaining validity is irrelevant. A dated record that expires is more honest than an undated one that never does.

Monitoring cannot silently renew it. Our sample records its continuous-evaluation status as not connected, and states the rule for when it is: connected monitoring may suspend a record on version drift or a material incident, but it cannot extend one. Renewal is an act, not a default.

04

Operationally

What does staleness mean operationally?

Re-run on the change, not on the calendar. If the six classes above are the trigger, then the rerun belongs in the pipeline that ships them — a failed critical scenario blocking a deploy the way any other failing test does. The alternative is discovering in a buyer's review that the version they are evaluating is not the version you tested.

→

Related

What else does a reviewer ask about this?

The questions in this bank overlap. A vendor who can answer one of these well can usually answer its neighbours badly, which is why they are tested together.

01What happens when a model swap changes behavior?Someone notices, and which someone is a design decision rather than luck: if the deployed version records only a model alias, the digest is a hash of a name and the swap is undetectable on the vendor's side, so the first observer is a customer running the case nobody wrote a test for.02What does it mean to freeze an agent version?Recording a digest of the four things that determine behavior — model, prompt, policy, and toolset — so any later change is detectable; a git commit is not enough, because two deploys of the same commit are different agents if the model alias resolved to a new checkpoint.03What evidence do enterprise reviewers accept about an AI agent?Executed runs against cases the vendor did not choose, on a frozen version, with the raw traces preserved and the results recomputable; operator-supplied evidence still counts as context, but it caps the conclusion, because no operator can upgrade its own evidence.04Does a refused request win on the second try?Sometimes. A refusal is a single-turn outcome; the same request re-entered in a new session or through a different channel meets an agent with no memory of refusing, so the question is whether the boundary is a property of the system or of one conversation.

The published bank is the visible subset. Buyer-specific questions and holdout scenarios are not published, because a question you can read in advance is a question you can train against. The rules that govern how any of these gets answered are public: how a conclusion is reached.

Answering it with evidence

Can you answer this about your own agent?

We keep the question bank, we run your frozen agent against it in a sandbox we control, and every failure lands on your desk before a buyer asks. Private, fixed fee, no buyer involvement. Send us the list of actions your agent can take and we will tell you which of these questions we would test first.

What a dry run needs, so nothing is a surprise later

  • An agent that takes real actions, not just answers questions
  • A version you can freeze for two weeks
  • A sandbox or endpoint we can reach
Book a 20-minute scoping call Or email your agent's action list