# Eve SOTA family evaluation and sample-design contract — 2026-09-13

**Task:** 12.2  
**Status:** preregistered and intentionally unmeasured  
**Machine record:**
[`eve-sota-evaluation-design/2026-09-13.json`](eve-sota-evaluation-design/2026-09-13.json)  
**Design
schema:**
[`eve-sota-evaluation-design.schema.json`](eve-sota-evaluation-design.schema.json)  
**Cohort
schema:**
[`eve-sota-evaluation-cohort.schema.json`](eve-sota-evaluation-cohort.schema.json)

## Decision

The prospective version-2 outcome scorecard is the numerical and governance
authority. Its operator-authorized amendment applies only to fresh cohorts
locked after that amendment; Task 12.2 does not change a target, floor, hard
lock, statistical unit, or decision owner.

For each of the exact fifteen Task 12.1 families, the generated record now
freezes:

- four independent case-diversity classes;
- deterministic, nested stochastic, and human run rules;
- separate pure, model-advisory, and human-authority rubric lanes;
- paired negative and benign controls;
- every applicable hard safety lock and its `HOLD` escalation;
- scorecard outcome and floor-criterion ownership;
- all seven required axis decisions, including explicit source-backed
  exclusions;
- every marginal stratum plus every plane × high/critical-risk intersection;
- a separate denominator for every floor criterion in every stratum.

The expansion contains 15 families, 774 strata, 206 criterion plans, and 10,585
criterion denominators. These are obligations, not results. Every client-facing
family conservatively includes all 25 platform ids ratified in the Task 0.8
charter inventory. Narrowing applicability requires a prospective amendment
before affected output is visible.

## Statistical rules

The statistical unit is copied verbatim from each Task 0.3 scorecard row. A case
id is the independent unit. Seeds, retries, turns, provider attempts, claims,
rubric items, and fault locations nested in that case never increase `n`.
Model-dependent cases require at least three fixed seeds, and every required
seed must pass.

For every Wilson criterion/stratum, the design requires at least 200 independent
units and reproduces the exact binomial probability of clearing the unchanged
two-sided 95% Wilson floor at the declared target. The semantic verifier rejects
an infeasible floor, a probability below 80%, or a claimed probability that it
cannot independently reproduce.

Continuous minima are deliberately not called power claims. Before a graded
cohort can lock, a separate non-promoting pilot must provide distribution, tail,
cluster, missingness, censoring, failure, and abstention assumptions. Each
criterion/stratum must preregister at least 10,000 bootstrap resamples, at least
10,000 power-simulation replicates, at least 80% planned probability of clearing
the floor, and the criterion's precision target. The exact graded `n` is the
maximum of the record's denominator minimum, scorecard composition minimum,
pilot-derived power size, and pilot-derived precision size. Pilot case ids may
not overlap graded case ids.

## Human authority

Every family names an actual human role for subjective labels and a separate
review role. The manifest must carry a privacy-safe but attributable human
assessor roster. The system under evaluation, evaluation harness, and model
judge may randomize or route-blind material, but they may not author, infer,
repair, or replace a missing human label. Model rubrics remain advisory until
their exact classes pass Task 12.3 calibration.

## Fail-closed evaluation CLI

Lock a cohort before any result:

```sh
node tools/eve-everywhere/evaluate-evaluation-cohort.mjs \
  --manifest=path/to/family.cohort-lock.json
```

Evaluate a result only against the separate immutable lock artifact:

```sh
node tools/eve-everywhere/evaluate-evaluation-cohort.mjs \
  --manifest=path/to/family.results.json \
  --lock=path/to/family.cohort-lock.json
```

The lock binds the design digest, exact primary case ids, nested seed ids,
criterion denominators, per-case strata, continuous pilot analyses, human
assessor roster, and zero amendments. Result admission compares the complete
plan projection to the separate lock and rejects post-result expansion or
rewriting.

The CLI also rejects missing or reordered denominators, undersized strata,
duplicate case ids, nested seed inflation, cross-stratum pooling, missing case
classes, missing or model-owned human assessment, overlapping/underpowered/
imprecise pilots, result sets not matching the locked case ids, failed required
seeds, hard-lock events, and aggregate-only results. A family-closed result
requires a human-owned label and content-hashed observation evidence for every
unit in every denominator; the CLI reads those evidence bytes and verifies their
hashes.

## Evidence and limitations

The generator pins byte-level receipts for the Task 12.2 policy, Task 0.3
scorecard, Task 12.1 taxonomy, ratified Task 0.8 charter inventory, threat-plane
inventory, model-leg inventory, and shipped-interface baseline. The verifier
independently reconstructs the family, axis, stratum, criterion, denominator,
hard-lock, Wilson-power, continuous-pilot, and ownership invariants.

Task 12.2 closes only the prospective design and manifest-admission machinery.
It records no pilot observation, product result, human label, model-quality
measurement, family pass, release admission, phase closure, or closure of G7 or
G14. Task 12.3 owns human/judge calibration; Task 12.4 owns long-horizon case
artifacts; Task 12.5 owns dataset governance; Task 12.7 owns release admission;
Task 12.8 owns charter-completeness decisions.