Status: not closed. This pack records completed repository work and the remaining external acquisitions. It is not evidence that any human author, learner, teacher, rater, or evaluator has been recruited.
Work-package charter#
- Task ID:
EVAL-002 - Owner: evaluation-data governance, V5 authored-case content, and V9 learning-research programs; repository implementation owner: Codex
- Audit finding: durable operator accept/reject rows and deterministic split helpers existed, but they did not establish best-authored quality, licenses, consent, exact artifact provenance, real runtime stages, rater governance, contamination freedom, or actual V5/V9 human cohorts
- Repository objective: prevent legacy feedback, fixtures, or planning manifests from being admitted as licensed human gold; preserve missing human work as owned, machine-readable acquisition blockers
- External objective: commission and contract the human-authored corpora and recruit the consented cohorts represented by the registry
- Rollback strategy: revert source commit
238ebf35f23959be140663567563a5401550e67e - Evidence-pack location: this file
Completed repository work#
The canonical LicensedHumanGoldCorpus record now carries:
- schema, corpus, version, seal, revocation, and manifest identity;
- best-authored positive, known-bad adversarial, borderline/high-disagreement, and pairwise-preference roles;
- optimization, development, and release-holdout splits;
- exact artifact bytes, author identity, provenance manifest, license receipt, consent proof, retention/deletion state, access policy, and audit reference;
- real provider/model, render, cook, in-engine, and placed evidence with exact provider, model, engine, build, environment, and placement versions;
- locales and cultural contexts per content family;
- versioned instructions, rubric, conflict policy, adjudication policy, rater expertise/locale/credentials, ratings, disagreement, and adjudications;
- human cohort recruitment, aggregate roster digest, consent, privacy, retention, access, independence, locale, expertise, and evidence;
- every optimization exposure and exact/near/semantic contamination analysis.
evaluateHumanGoldCorpusReadiness compares a corpus with a versioned coverage
policy. It requires best-authored, borderline, and release-holdout items for
every content family; known-bad items for every hard gate; per-family locale and
cultural coverage; configured real-output stages; active rights, consent,
retention, access, and rater governance; clean holdout contamination coverage;
and configured item/cohort acquisitions. assertHumanGoldCorpusReady raises a
typed error containing every exact blocker.
The contract/Prisma alignment table explicitly records the corpus as
code_only. Existing CrossCuttingGoldSetEntry rows remain valid operator
feedback, but no persistence projection falsely upgrades them into licensed
corpus, artifact, rater, cohort, exposure, or contamination records.
Optimization and release admission#
The Agentic Studio admission boundary recomputes readiness on every request; a caller cannot substitute a stale passing report. Purpose determines the only eligible split:
| Purpose | Admissible split |
|---|---|
| Model, prompt, rubric, policy, or judge optimization | optimization |
| Development evaluation | development |
| Release evaluation | release_holdout |
Explicit wrong-split, duplicate, unknown, or access-unauthorized requests fail with typed errors. Every successful admission binds the exact corpus manifest, policy, consumer version, item IDs, artifact digests, and selection digest, and emits exposure records that the caller must append atomically. Release holdouts cannot silently enter model training, prompt/rubric/policy optimization, judge calibration, or development evaluation.
Machine-readable external acquisition register#
evaluation/human-gold/acquisition-registry.v1.json contains six acquisition
requests and deliberately records zero acquired units:
| Request | Required | Acquired | Status |
|---|---|---|---|
| Portfolio best-authored content-family corpus program | 1 corpus | 0 | Blocked |
| Portfolio hard-gate adversarial corpus program | 1 corpus | 0 | Blocked |
| Structured V5 authored-case corpus | 25 items | 0 | Blocked |
| V9 real learner cohort | 1 cohort | 0 | Blocked |
| V9 practicing-teacher cohort | 1 cohort | 0 | Blocked |
| V9 independent evaluator cohort | 1 cohort | 0 | Blocked |
For cohort requests, the registry counts cohort acquisitions, not participants. The eventual coverage policy must set participant minima from an approved, powered study protocol. No sample size, contract, evidence receipt, or participant identity is fabricated here.
The blockers require, among other things, commissioned human authors, executed rights/consent/data-processing terms, structured/cooked/placed/playtested V5 cases, approved sampling and study protocols, participant privacy/withdrawal controls, target-learner and practicing-teacher recruitment, and an evaluator independent of the optimization team. These actions require external authority and coordination beyond a repository mutation.
Adversarial fixture matrix#
| Fixture | Result |
|---|---|
| Provider/runtime output labeled best-authored | Schema rejects the substituted baseline |
| Known-bad item names no expected hard-gate failure | Schema rejects the item |
| Borderline item lacks measured high disagreement/adjudication | Schema rejects the item |
| Provider, engine, build, or placement version is absent | Stage evidence is invalid |
| Rights or consent proof names another artifact | Corpus schema rejects cross-subject substitution |
| Rating or adjudication names another item | Corpus schema rejects cross-item substitution |
| Rights expire, data is deleted, access is suspended, or a rater conflicts | Readiness is blocked with exact subjects |
| Release artifact digest appears in an optimization split | Readiness is blocked for holdout overlap |
| Holdout is exposed to prompt optimization | Readiness and admission block the exposure |
| Holdout analysis is missing or contaminated | Readiness is blocked |
| V5/V9 cohort or item requirement is absent | Readiness is blocked; no fixture satisfies it |
| Caller requests unknown, duplicate, or wrong-split items | Admission fails with a typed error |
| Caller presents a draft corpus | Admission recomputes readiness and fails |
| Registry request claims acquisition without full count, contract, and evidence | Registry schema rejects the claim |
Automated verification#
The integrated repository work passed 4,390/4,390 non-duplicated owning package tests:
| Surface | Result |
|---|---|
| Canonical contracts full package, including all migration/rollback cases | 4,053/4,053 |
| Agentic Studio full package, including purpose/split admission | 337/337 |
The new coverage consists of 15 contracts/acquisition-registry tests and 7
Agentic Studio admission tests. Production library typechecks passed for both
packages. Targeted Nx builds passed for @oshun/contracts and
@oshun/agentic-studio, including the latter's six build dependencies. Explicit
ESLint over every changed TypeScript file passed with zero warnings; Prettier
and git diff --check passed.
Representative commands:
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm --dir libs/oshun/agentic-studio exec vitest run
pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/oshun/agentic-studio/tsconfig.lib.json --noEmit
NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
NX_TUI=false pnpm nx run @oshun/agentic-studio:build --parallel=1 --skip-nx-cache
pnpm exec eslint <all changed TypeScript files> --max-warnings=0 --no-warn-ignored
pnpm exec prettier --check <all changed non-lock files>
git diff --check
Bounded verification notes#
- The first full contracts run passed 4,049 tests but four Prisma migration cases exceeded their fixed 30-second timeout while another workspace agent's Nx graph workers saturated CPU. The exact migration suite then passed 9/9 with a 120-second ceiling, and the complete contracts suite passed all 175 files and 4,053 tests under the same supervised ceiling.
- Contracts still emit the repository's existing missing
libs/contracts/src/index.js.mapwarning; it does not affect execution. - The first Agentic Studio Nx build exposed that dependency-output resolution loses callback inference for the new Zod-derived types. Explicit type annotations preserve the same source types; the rerun compiled all seven build targets successfully.
- The source commit used the hook's documented
SKIP_TYPECHECK=1after both production typechecks, full package suites, and targeted builds passed. Common-contract changes otherwise expand the hook into an unsafe broad workspace typecheck on this host. Hook lint, formatting, asset, stub, and documentation gates passed. - Verification ran sequentially on a 15 GiB RAM/31 GiB swap host while another agent held active Nx workers. Available memory remained above 9 GiB before the full gates.
- The unrelated working
pnpm-lock.yamlblob was not staged and was preserved across the source rebase. - No customer UI, PWA, admin, mobile, Unreal, or visual surface changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.
Artifact identity#
| Artifact | SHA-256 |
|---|---|
| Licensed corpus/readiness contract | 186d7026111c70472da2c267e12369446370ba67606e02a5fa747965a8a72e7e |
| Purpose/split admission boundary | 68b94b6f1f4e4b8efd53869c1a1bb10328ed488e50b430e78364347e9dd54605 |
| External acquisition registry | 52e5782f5e6fe177a6977bdda21cbdef208bf74f50fd5c8f0212c7c6e9691ae7 |
Remaining work and authority boundary#
All eight EVAL-002 ledger boxes remain open. Closing them requires actual licensed and consented human work—not more fixtures:
- Approve the complete content-family and hard-gate sampling frames.
- Contract and compensate qualified human authors and raters under usable rights, provenance, retention, access, deletion, and withdrawal terms.
- Produce, adjudicate, and seal positive, adversarial, borderline, culturally/linguistically diverse, and preference items.
- Reserve release holdouts before prompt/rubric/policy/model optimization and run contamination analysis against every optimization source bundle.
- Structure, cook, place, and play all 25 V5 cases with exact author and rights evidence.
- Approve a powered V9 human-participant protocol, recruit real learners, practicing teachers, and independent evaluators, and capture consented outcome evidence.
The assistant has no authority to contract, compensate, recruit, consent, or message those people. Repository work now makes that missing authority visible and prevents release or optimization from bypassing it.
Publication identity#
- Repository source commit:
238ebf35f23959be140663567563a5401550e67e - Repository source publication tip:
238ebf35f23959be140663567563a5401550e67e - EVAL-002 closure evidence commit: not applicable; EVAL-002 is open