- Status: Accepted
- Date: 2026-09-13
- Decision owner: Governance Lead
- Operational owners: Evaluation Science Lead, Privacy Lead, Security Lead, QA Lead
Context#
Eve evaluation assets previously carried local provenance and held-out controls in several harnesses, but no total registry enforced source/licence, privacy and redaction, content hashes, split separation, contamination, freshness, retirement, ownership, and evidence retention together. This also made it too easy to confuse regression cases, provider output, machine-proposed labels, and synthetic contract fixtures with independent graded evidence.
Decision#
All tracked Eve eval-data candidates in the bounded assistant-eval and audit namespaces are registered exactly once. Admission is by dataset and declared use and fails closed on an unregistered path, byte or Git-blob drift, unresolved licence, incomplete privacy review, stale review, incompatible use, overlap, missing owner, or weakened retention.
Train, tune, graded, and held-out are mutually exclusive promotion splits. Evaluator-only material is never candidate-visible. Exposure of held-out material requires retirement and replacement. Repository-authored assets retain Apache-2.0 provenance; provider-generated output without recorded rights stays quarantined and cannot be training, calibration, graded, held-out, or promotion evidence. Machine-authored label proposals never count as human evidence.
Raw execution evidence relied upon by an evidence manifest is content-hashed and retained for at least 730 days. It is superseded by a new record rather than rewritten. Credentials are never retained; sensitive content must be redacted or encrypted before retention.
Consequences#
The ordinary assistant deck is explicitly tuning/regression material. The engineering benchmark is the only currently promotion-eligible held-out dataset in this bounded inventory. The assistant held-out drift set remains non-gating by its existing contract. SMX judge transcripts and live operator-memory outputs remain visible but quarantined until their independent human and rights reviews are complete. Synthetic trajectory fixtures remain nonpromoting contract tests.
This decision establishes governance machinery and truthful status; it does not complete Task 12.3, validate a judge, prove dataset quality, or close G7/G14 or Phase 12.