Task 12.5 is implemented as a generated, byte-bound registry plus a fail-closed semantic verifier and per-use admission CLI. The governing decision is ADR-0085.
What is governed#
The registry independently discovers and accounts for every tracked Eve
evaluation case, corpus, label sheet, measured response set, evaluator oracle,
and synthetic contract fixture in the bounded
apps/oshun/bff/src/assistant/evals and named docs/audits namespaces. Each
artifact has byte length, SHA-256, Git blob identity, dataset version, source
and licence posture, privacy/redaction status, split and allowed use,
contamination rules, review/expiry/retirement policy, accountable roles, and
retention requirements.
The registry separates tuning/regression, held-out, graded, calibration, reference, evaluator-only, and contract-fixture material. A use must be explicitly allowed. Held-out material may not overlap or permit train/tune use; evaluator-only oracles remain candidate-hidden. Promotion additionally requires an admitted, current, exactly hashed, licence-approved, privacy-reviewed, promotion-eligible dataset.
Honest holds#
- The ordinary deck and operator-memory regression cases are development/tuning evidence, never independent graded or held-out evidence.
- The assistant held-out deck remains a non-gating drift set under its existing harness contract.
- SMX provider transcripts are quarantined until provider-output rights and the actual blinded human labels are recorded. Task 12.3 remains open.
- Operator-memory live output is quarantined because provider-output rights are unresolved and its proposed labels explicitly say they are machine-authored and ineligible as human evidence.
- The docs relevance record remains quarantined as graded evidence while five semantic classes await human labels.
- Synthetic trajectory fixtures prove contracts only and cannot promote a model or family.
Verification boundary#
The verifier checks total inventory ownership, live hashes, Git blobs, source/licence, privacy/redaction, split isolation, known label holds, sensitive-secret patterns, freshness, ownership, retention, and exact Task 12.5 bindings to the NIST GenAI IP/media and NIST AML poisoning crosswalk entries. Targeted negative controls demonstrate rejection of omissions, hash/version drift, unresolved licences, privacy bypass, personal-data admission, split overlap, held-out tuning, quarantine promotion, expiry, missing owners, weakened evidence retention, crosswalk drift, and machine-label promotion.
Passing these checks proves the governance contract and current registry state only. It does not prove evaluator validity, human agreement, real model quality, or release admission.