Disciplines · Audits

Eve SOTA long-horizon trajectory grading — September 2026

Task 12.4 adds an event-level, pre-result-locked trajectory grader.

7sections3 minread
Markdown source
On this page

Decision#

Task 12.4 adds an event-level, pre-result-locked trajectory grader. A correct or polished final answer is not sufficient: the grader separately decides goal completion, unnecessary steps, recovery, repeated error, terminal state drift, premature success, safe abstention, and evidence fidelity. The cohort CLI binds the exact Task 12.2 design digest, exact case-policy lock, exact case inventory, and a result timestamp strictly after the lock.

This closes grading machinery, not a product-family result. The committed test fixtures are synthetic contract cases. They do not show that Eve completed a real long-horizon task, met a family floor, or qualifies for release.

Source inspection#

  • The Task 12.2 design record defines fifteen evaluation families and requires trajectory-level evidence, but it deliberately left this grader to Task 12.4.
  • The assistant battery records final text, tool calls, completion, and provider metadata. It does not independently grade excess actions, fault recurrence, recovery order, terminal state, or evidence bytes.
  • The generic cohort CLI protects sample design and denominators. It does not interpret an ordered agent trajectory. This task therefore adds a separate lock and result contract rather than overloading final-answer grades.

Eight independent dimensions#

Dimension Pre-result authority Fail-closed observation
Goal completion Required milestone ids and completed terminal Every milestone must occur before the terminal
Unnecessary steps Per-action maximum occurrence budget with rationale Unknown or over-budget action, including recovery action, fails
Recovery Fault recoverability and required recovery action Recoverable fault must have a later successful, matching recovery
Repeated error Maximum occurrences per fault signature Same-signature recurrence above its frozen maximum fails
State drift Required terminal key/value invariants Last observed value for every required key must match
Premature success Milestones, resolved recoverable faults, and faithful evidence A completed terminal before any prerequisite fails
Safe abstention Allowed reason and required observed fault signatures Unsupported abstention fails; supported abstention is honest but not success
Evidence fidelity Evidence id, expected SHA-256, and supported claim ids Missing, changed, uncited, or unseen evidence fails

The report uses three decisions. passed is the only promotion-eligible decision. safely-abstained is an honest terminal state but keeps goal completion failed and is never counted as success. failed covers every other case. One failing dimension holds the case even if the terminal says completed.

Interfaces#

The lock, results, and report schemas are:

  • docs/audits/eve-sota-trajectory-lock.schema.json
  • docs/audits/eve-sota-trajectory-results.schema.json
  • docs/audits/eve-sota-trajectory-report.schema.json

Grade an already locked cohort with:

sh
node tools/eve-everywhere/grade-trajectory-cohort.mjs \
  --lock=path/to/trajectory-lock.json \
  --results=path/to/trajectory-results.json \
  --output=path/to/trajectory-report.json

The CLI exits 0 only when every locked case is passed, 2 for an honest graded HOLD, and nonzero on malformed, changed, post-result, missing, or unexpected cohort data.

Frozen grader-performance check#

Before measuring, the deterministic benchmark is fixed at 1,000 warm-up grades and four measured batches of 5,000 independent six-event synthetic grades. The worst batch must sustain at least 5,000 trajectory grades per second and process maximum-RSS growth must not exceed 128 MiB. The retained measurement is only grader-overhead evidence; it is not agent latency, model quality, provider latency, or a product trajectory.

Negative controls#

The semantic verifier plants and observes failures for a final-answer-only mask, an excess step, unresolved recovery, repeated error, state drift, premature success, unjustified abstention, changed evidence, a post-result lock, lock drift, a missing case, an unlocked case, a dropped grading dimension, and promotion of safe abstention. A final green run follows every red control.

Remaining boundaries#

  • Task 12.3 remains open until a person supplies the 42 blinded labels and the per-class judge tournament is executed. This task does not infer them.
  • Task 12.5 owns data source, licence, privacy, split, contamination, freshness, retirement, and retention governance.
  • Task 12.7 owns release admission and Task 12.8 owns charter completeness.
  • No actual evaluation cohort or product result is created here. Future trajectory producers must emit facts from their runtime and bind their own pre-result policies; they may not copy the synthetic fixture as evidence.