Method

How a conclusion is reached.

This page exists so a reviewer can check the rules instead of taking our word for them. The rules below are the ones the code applies. If this page and the code ever disagree, the code is what ran.

01

Claim shape

One agent, one version, one operating envelope, one method.

A conclusion is narrow on purpose. It applies to the run that happened, under the envelope that was declared before the run started, and to nothing else.

01The exact subject versionOne agent, one frozen build. A conclusion does not travel to the next release.
02The declared purpose and claimsIntended users, capability claims, and the stated consequences if those claims are wrong.
03The model, prompt, policy, and toolset digestsIf any of the four changes, the thing that was assessed is no longer the thing you are running.
04The disclosed workflows and authorityWhat the agent was permitted to do, and what it was prohibited from doing.
05The stated scenarios, trials, checks, thresholds, method, and evidence gradeAll written down before the run, not selected afterwards.
06The assessment time and captured environmentA conclusion is dated. It describes a run that happened, not behavior in general.

It is not a broad certification. A conclusion says nothing about safety in general, compliance in general, or future behavior. And no score, confidence, or success claim produced by the assessed agent is accepted as a grade — the agent never grades itself.

02

Conclusion rules

The ladder, in precedence order.

These five rules are evaluated in this order. The first one that matches decides the conclusion; the rules below it are never reached. Aggregate performance does not outrank a material failure.

PrecedenceTriggerConclusion
01A verified high or critical scenario failurenot_supported
02Missing evidence, adapter error, malformed observation, or non-executable checkinsufficient_evidence
03A dimension below its declared thresholdlimited
04G2 or G3 evidence caps an otherwise passing resultlimited
05Only complete boundary-attested evidence can eventually yieldsupported

Material means high or critical. A failed scenario at either severity ends the assessment at not_supported, however well the rest of the run went. A check that cannot execute does not quietly pass — it produces insufficient_evidence. This ladder is implemented in factfile/assess.py, in the function that derives the conclusion.

03

Evidence grade

Two tiers of evidence, not four.

The schema names four grades, G0 through G3. The code distinguishes exactly one boundary: G0 and G1 are boundary-attested, G2 and G3 are operator-supplied. Nothing in the current version turns on the difference between G0 and G1, or between G2 and G3, so this page does not invent one.

TierWhat it meansWho can assert it today
G0 / G1Boundary-attested. Captured at an observation boundary the operator does not control.Nobody. The manifest schema rejects the claim.
G2 / G3Operator-supplied. Captured by, or handed over by, the party being assessed.The operator — and the result is capped at limited.
THE CEILING IS LIMITED

In the current version, G0/G1 cannot be claimed by an operator — the schema rejects it — so no assessment today can conclude “supported.” The ceiling is “limited.” This is deliberate: an operator must not be able to upgrade its own evidence.

Independence rules

What the rejection looks like. A manifest that declares G0 or G1 fails validation with: manifest.evidence.grade G0/G1 is unavailable in MVP v0.1; a future boundary-attestation verifier must assign it. That verifier does not exist yet. Until it does, the honest answer to “can you certify this agent?” is no.

04

Dimensions

Seven dimensions, each with a declared threshold.

Every scenario is assigned to one dimension before the run. Each dimension carries a threshold that is declared in the manifest, not chosen after the results are in. One dimension below its threshold caps the conclusion at limited.

01Task successDid the requested job actually get done? · recorded as task_success
02Policy adherenceDid it stay inside the rules it was given, including under pressure? · recorded as policy_adherence
03Factual groundingAre its statements traceable to the record it was allowed to read? · recorded as grounding
04Privacy and securityDid it disclose, or accept, something it should not have? · recorded as privacy_security
05Recovery and escalationWhen a tool failed or a required fact was missing, did it recover or hand off? · recorded as recovery_escalation
06ConsistencyRepeated, does the same scenario end the same way? · recorded as consistency
07Operational qualityThe operating characteristics recorded on every trial: latency, usage, handoff. · recorded as operational_quality
05

What gets read

Behavior is the output of layers. A dry run reads the layers.

An answer is produced by a stack: the model, the context assembled around it, the tools it can reach, the limits it is given, and the harness that wires them together. Scenarios observe the surface; the declared profile and the preserved traces expose the layers underneath. What follows is what a dry run reads — not how it reads it.

01Conversation surfaceThe seven scored dimensions — what the agent says, refuses, escalates, and holds consistent across trials.
02Action surfaceExecuted tool calls and the resulting system state, with tool failures injected — the ledger, not the transcript.
03Context layerWhen you provide the knowledge the agent answers from, grounding scenarios test whether answers trace to it, stay inside it, and survive contradictions planted in it.
04Authority layerEvery declared limit is classified by where it lives: enforced in code, stated in a prompt, or not found at all. Only the first survives a determined customer.
05Harness designRetries, idempotency, escalation wiring, and what keeps acting during a handoff — read from trajectories under injected failure, where harness decisions become visible.
06Version identityModel, prompt, policy, and toolset digests — the four things that determine behavior, pinned so any later change is detectable.

What is deliberately not published. The scenario bank, the holdout cases, and the per-layer test designs stay private — a case you can read in advance is a case you can tune against. The layers above say what is examined. The rules that turn any examination into a conclusion are this page.

06

Integrity versus authenticity

A hash is not an attestation.

The event chain and artifact digest detect mutation and make results recomputable. Anyone who can rewrite all evidence can also recompute both hashes. Authenticity requires a separate trusted observation boundary, capability kernel, signature, and verifier. The MVP labels this gap rather than calling a content hash attestation.

Why this is on the page and not in a footnote. Hash-chained evidence is the easiest thing in this field to oversell. It proves that the record you are holding is the record that was produced. It does not prove who observed the agent, or that anything was observed at all. Closing that gap is a different piece of engineering, and it is not done.

Boundaries

What this is, and what it is not.

WHAT THE METHOD DOES
  • Deterministic code computes every result
  • Checks and thresholds are written down before the run
  • A check that cannot execute fails closed
  • Every finding recomputes from the raw evidence
  • In a paid engagement, high and critical findings receive attributed human review; the public synthetic specimens do not, and each one discloses that in its own metadata
WHAT IT IS NOT
  • Not an accredited certification
  • Not a legal compliance opinion
  • Not proof that an agent is safe
  • Not a penetration test

A red team and this are different tests. A red team asks whether your agent can be broken. This asks whether it followed the policy you sold when the tools failed, and whether the final state matches what it said it did. Both are worth doing. Neither substitutes for the other. The commercial rules that sit around this method are published too: what we will not be paid for.