Eve library

EVAL-002 progress evidence — licensed human-gold acquisition boundary

Evidence

Revision 1

On this page

Status: not closed. This pack records completed repository work and the remaining external acquisitions. It is not evidence that any human author, learner, teacher, rater, or evaluator has been recruited.

Work-package charter#

  • Task ID: EVAL-002
  • Owner: evaluation-data governance, V5 authored-case content, and V9 learning-research programs; repository implementation owner: Codex
  • Audit finding: durable operator accept/reject rows and deterministic split helpers existed, but they did not establish best-authored quality, licenses, consent, exact artifact provenance, real runtime stages, rater governance, contamination freedom, or actual V5/V9 human cohorts
  • Repository objective: prevent legacy feedback, fixtures, or planning manifests from being admitted as licensed human gold; preserve missing human work as owned, machine-readable acquisition blockers
  • External objective: commission and contract the human-authored corpora and recruit the consented cohorts represented by the registry
  • Rollback strategy: revert source commit 238ebf35f23959be140663567563a5401550e67e
  • Evidence-pack location: this file

Completed repository work#

The canonical LicensedHumanGoldCorpus record now carries:

  • schema, corpus, version, seal, revocation, and manifest identity;
  • best-authored positive, known-bad adversarial, borderline/high-disagreement, and pairwise-preference roles;
  • optimization, development, and release-holdout splits;
  • exact artifact bytes, author identity, provenance manifest, license receipt, consent proof, retention/deletion state, access policy, and audit reference;
  • real provider/model, render, cook, in-engine, and placed evidence with exact provider, model, engine, build, environment, and placement versions;
  • locales and cultural contexts per content family;
  • versioned instructions, rubric, conflict policy, adjudication policy, rater expertise/locale/credentials, ratings, disagreement, and adjudications;
  • human cohort recruitment, aggregate roster digest, consent, privacy, retention, access, independence, locale, expertise, and evidence;
  • every optimization exposure and exact/near/semantic contamination analysis.

evaluateHumanGoldCorpusReadiness compares a corpus with a versioned coverage policy. It requires best-authored, borderline, and release-holdout items for every content family; known-bad items for every hard gate; per-family locale and cultural coverage; configured real-output stages; active rights, consent, retention, access, and rater governance; clean holdout contamination coverage; and configured item/cohort acquisitions. assertHumanGoldCorpusReady raises a typed error containing every exact blocker.

The contract/Prisma alignment table explicitly records the corpus as code_only. Existing CrossCuttingGoldSetEntry rows remain valid operator feedback, but no persistence projection falsely upgrades them into licensed corpus, artifact, rater, cohort, exposure, or contamination records.

Optimization and release admission#

The Agentic Studio admission boundary recomputes readiness on every request; a caller cannot substitute a stale passing report. Purpose determines the only eligible split:

Purpose Admissible split
Model, prompt, rubric, policy, or judge optimization optimization
Development evaluation development
Release evaluation release_holdout

Explicit wrong-split, duplicate, unknown, or access-unauthorized requests fail with typed errors. Every successful admission binds the exact corpus manifest, policy, consumer version, item IDs, artifact digests, and selection digest, and emits exposure records that the caller must append atomically. Release holdouts cannot silently enter model training, prompt/rubric/policy optimization, judge calibration, or development evaluation.

Machine-readable external acquisition register#

evaluation/human-gold/acquisition-registry.v1.json contains six acquisition requests and deliberately records zero acquired units:

Request Required Acquired Status
Portfolio best-authored content-family corpus program 1 corpus 0 Blocked
Portfolio hard-gate adversarial corpus program 1 corpus 0 Blocked
Structured V5 authored-case corpus 25 items 0 Blocked
V9 real learner cohort 1 cohort 0 Blocked
V9 practicing-teacher cohort 1 cohort 0 Blocked
V9 independent evaluator cohort 1 cohort 0 Blocked

For cohort requests, the registry counts cohort acquisitions, not participants. The eventual coverage policy must set participant minima from an approved, powered study protocol. No sample size, contract, evidence receipt, or participant identity is fabricated here.

The blockers require, among other things, commissioned human authors, executed rights/consent/data-processing terms, structured/cooked/placed/playtested V5 cases, approved sampling and study protocols, participant privacy/withdrawal controls, target-learner and practicing-teacher recruitment, and an evaluator independent of the optimization team. These actions require external authority and coordination beyond a repository mutation.

Adversarial fixture matrix#

Fixture Result
Provider/runtime output labeled best-authored Schema rejects the substituted baseline
Known-bad item names no expected hard-gate failure Schema rejects the item
Borderline item lacks measured high disagreement/adjudication Schema rejects the item
Provider, engine, build, or placement version is absent Stage evidence is invalid
Rights or consent proof names another artifact Corpus schema rejects cross-subject substitution
Rating or adjudication names another item Corpus schema rejects cross-item substitution
Rights expire, data is deleted, access is suspended, or a rater conflicts Readiness is blocked with exact subjects
Release artifact digest appears in an optimization split Readiness is blocked for holdout overlap
Holdout is exposed to prompt optimization Readiness and admission block the exposure
Holdout analysis is missing or contaminated Readiness is blocked
V5/V9 cohort or item requirement is absent Readiness is blocked; no fixture satisfies it
Caller requests unknown, duplicate, or wrong-split items Admission fails with a typed error
Caller presents a draft corpus Admission recomputes readiness and fails
Registry request claims acquisition without full count, contract, and evidence Registry schema rejects the claim

Automated verification#

The integrated repository work passed 4,390/4,390 non-duplicated owning package tests:

Surface Result
Canonical contracts full package, including all migration/rollback cases 4,053/4,053
Agentic Studio full package, including purpose/split admission 337/337

The new coverage consists of 15 contracts/acquisition-registry tests and 7 Agentic Studio admission tests. Production library typechecks passed for both packages. Targeted Nx builds passed for @oshun/contracts and @oshun/agentic-studio, including the latter's six build dependencies. Explicit ESLint over every changed TypeScript file passed with zero warnings; Prettier and git diff --check passed.

Representative commands:

text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm --dir libs/oshun/agentic-studio exec vitest run
pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit
pnpm exec tsc -p libs/oshun/agentic-studio/tsconfig.lib.json --noEmit
NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
NX_TUI=false pnpm nx run @oshun/agentic-studio:build --parallel=1 --skip-nx-cache
pnpm exec eslint <all changed TypeScript files> --max-warnings=0 --no-warn-ignored
pnpm exec prettier --check <all changed non-lock files>
git diff --check

Bounded verification notes#

  • The first full contracts run passed 4,049 tests but four Prisma migration cases exceeded their fixed 30-second timeout while another workspace agent's Nx graph workers saturated CPU. The exact migration suite then passed 9/9 with a 120-second ceiling, and the complete contracts suite passed all 175 files and 4,053 tests under the same supervised ceiling.
  • Contracts still emit the repository's existing missing libs/contracts/src/index.js.map warning; it does not affect execution.
  • The first Agentic Studio Nx build exposed that dependency-output resolution loses callback inference for the new Zod-derived types. Explicit type annotations preserve the same source types; the rerun compiled all seven build targets successfully.
  • The source commit used the hook's documented SKIP_TYPECHECK=1 after both production typechecks, full package suites, and targeted builds passed. Common-contract changes otherwise expand the hook into an unsafe broad workspace typecheck on this host. Hook lint, formatting, asset, stub, and documentation gates passed.
  • Verification ran sequentially on a 15 GiB RAM/31 GiB swap host while another agent held active Nx workers. Available memory remained above 9 GiB before the full gates.
  • The unrelated working pnpm-lock.yaml blob was not staged and was preserved across the source rebase.
  • No customer UI, PWA, admin, mobile, Unreal, or visual surface changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Artifact identity#

Artifact SHA-256
Licensed corpus/readiness contract 186d7026111c70472da2c267e12369446370ba67606e02a5fa747965a8a72e7e
Purpose/split admission boundary 68b94b6f1f4e4b8efd53869c1a1bb10328ed488e50b430e78364347e9dd54605
External acquisition registry 52e5782f5e6fe177a6977bdda21cbdef208bf74f50fd5c8f0212c7c6e9691ae7

Remaining work and authority boundary#

All eight EVAL-002 ledger boxes remain open. Closing them requires actual licensed and consented human work—not more fixtures:

  1. Approve the complete content-family and hard-gate sampling frames.
  2. Contract and compensate qualified human authors and raters under usable rights, provenance, retention, access, deletion, and withdrawal terms.
  3. Produce, adjudicate, and seal positive, adversarial, borderline, culturally/linguistically diverse, and preference items.
  4. Reserve release holdouts before prompt/rubric/policy/model optimization and run contamination analysis against every optimization source bundle.
  5. Structure, cook, place, and play all 25 V5 cases with exact author and rights evidence.
  6. Approve a powered V9 human-participant protocol, recruit real learners, practicing teachers, and independent evaluators, and capture consented outcome evidence.

The assistant has no authority to contract, compensate, recruit, consent, or message those people. Repository work now makes that missing authority visible and prevents release or optimization from bypassing it.

Publication identity#

  • Repository source commit: 238ebf35f23959be140663567563a5401550e67e
  • Repository source publication tip: 238ebf35f23959be140663567563a5401550e67e
  • EVAL-002 closure evidence commit: not applicable; EVAL-002 is open