# Eve SOTA long-horizon trajectory grading — September 2026

## Decision

Task 12.4 adds an event-level, pre-result-locked trajectory grader. A correct or
polished final answer is not sufficient: the grader separately decides goal
completion, unnecessary steps, recovery, repeated error, terminal state drift,
premature success, safe abstention, and evidence fidelity. The cohort CLI binds
the exact Task 12.2 design digest, exact case-policy lock, exact case inventory,
and a result timestamp strictly after the lock.

This closes grading machinery, not a product-family result. The committed test
fixtures are synthetic contract cases. They do not show that Eve completed a
real long-horizon task, met a family floor, or qualifies for release.

## Source inspection

- The Task 12.2 design record defines fifteen evaluation families and requires
  trajectory-level evidence, but it deliberately left this grader to Task 12.4.
- The assistant battery records final text, tool calls, completion, and provider
  metadata. It does not independently grade excess actions, fault recurrence,
  recovery order, terminal state, or evidence bytes.
- The generic cohort CLI protects sample design and denominators. It does not
  interpret an ordered agent trajectory. This task therefore adds a separate
  lock and result contract rather than overloading final-answer grades.

## Eight independent dimensions

| Dimension         | Pre-result authority                                           | Fail-closed observation                                                      |
| ----------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Goal completion   | Required milestone ids and `completed` terminal                | Every milestone must occur before the terminal                               |
| Unnecessary steps | Per-action maximum occurrence budget with rationale            | Unknown or over-budget action, including recovery action, fails              |
| Recovery          | Fault recoverability and required recovery action              | Recoverable fault must have a later successful, matching recovery            |
| Repeated error    | Maximum occurrences per fault signature                        | Same-signature recurrence above its frozen maximum fails                     |
| State drift       | Required terminal key/value invariants                         | Last observed value for every required key must match                        |
| Premature success | Milestones, resolved recoverable faults, and faithful evidence | A `completed` terminal before any prerequisite fails                         |
| Safe abstention   | Allowed reason and required observed fault signatures          | Unsupported abstention fails; supported abstention is honest but not success |
| Evidence fidelity | Evidence id, expected SHA-256, and supported claim ids         | Missing, changed, uncited, or unseen evidence fails                          |

The report uses three decisions. `passed` is the only promotion-eligible
decision. `safely-abstained` is an honest terminal state but keeps goal
completion failed and is never counted as success. `failed` covers every other
case. One failing dimension holds the case even if the terminal says
`completed`.

## Interfaces

The lock, results, and report schemas are:

- `docs/audits/eve-sota-trajectory-lock.schema.json`
- `docs/audits/eve-sota-trajectory-results.schema.json`
- `docs/audits/eve-sota-trajectory-report.schema.json`

Grade an already locked cohort with:

```sh
node tools/eve-everywhere/grade-trajectory-cohort.mjs \
  --lock=path/to/trajectory-lock.json \
  --results=path/to/trajectory-results.json \
  --output=path/to/trajectory-report.json
```

The CLI exits `0` only when every locked case is `passed`, `2` for an honest
graded HOLD, and nonzero on malformed, changed, post-result, missing, or
unexpected cohort data.

## Frozen grader-performance check

Before measuring, the deterministic benchmark is fixed at 1,000 warm-up grades
and four measured batches of 5,000 independent six-event synthetic grades. The
worst batch must sustain at least 5,000 trajectory grades per second and process
maximum-RSS growth must not exceed 128 MiB. The retained measurement is only
grader-overhead evidence; it is not agent latency, model quality, provider
latency, or a product trajectory.

## Negative controls

The semantic verifier plants and observes failures for a final-answer-only mask,
an excess step, unresolved recovery, repeated error, state drift, premature
success, unjustified abstention, changed evidence, a post-result lock, lock
drift, a missing case, an unlocked case, a dropped grading dimension, and
promotion of safe abstention. A final green run follows every red control.

## Remaining boundaries

- Task 12.3 remains open until a person supplies the 42 blinded labels and the
  per-class judge tournament is executed. This task does not infer them.
- Task 12.5 owns data source, licence, privacy, split, contamination, freshness,
  retirement, and retention governance.
- Task 12.7 owns release admission and Task 12.8 owns charter completeness.
- No actual evaluation cohort or product result is created here. Future
  trajectory producers must emit facts from their runtime and bind their own
  pre-result policies; they may not copy the synthetic fixture as evidence.