Status: partially complete, not closed. The repository now prevents lexical diversity and uncalibrated/PCA model scores from making release or search decisions. It does not claim that any real human-gold calibration has occurred. EVAL-002 still records zero acquired corpus/cohort units.
Work-package charter#
- Task ID:
EVAL-003 - Owner: evaluation governance, model-judge calibration, creative-autonomy quality, and V8/V10 case release; repository implementation owner: Codex
- Audit finding: position-mitigated pairwise judging and several isolated calibration statistics existed, but no canonical record bound heterogeneous judge identities, human holdouts, bias probes, metrics, routing, approvals, and change regression. Creative-autonomy production still selected with lexical similarity, while V8's batch-dependent PCA/heuristic G5 verdict was a hard release gate.
- Repository objective: make uncalibrated judge scores and lexical/PCA diversity diagnostic-only; fail closed unless exact, current, human-approved calibration and validated semantic evidence are present.
- External objective: acquire the EVAL-002 human gold, recruit independent raters, run the blinded calibration study and bias probes, and issue the first real approved certificates.
- Rollback strategy: revert source commit
47438840440d658eeb7f6b4185fcb1ed68f384fb. - Evidence-pack location: this file.
Canonical judge-calibration evidence#
ModelJudgeCalibrationCertificate and its supporting records now require:
- exact judge, provider, provider-version, model, model-version, model-family, prompt digest/version, rubric version, policy version, and response-privacy identity;
- declared provider/model-family minima, computed diversity shortfall, and an explicit degraded reason; degraded diversity cannot receive release approval;
- blinded aliases, sealed identity-map digest, both candidate orders, both raw order picks, per-order confidence/evidence, and a winner derived from order agreement rather than claimed by the caller;
- position, verbosity, style, self-family, reference-leakage, contamination, and judge-pleasing bias probes, with exact heldout items and threshold-derived pass/fail;
- per-item, per-dimension, per-judge human observations with multiple human rating references, confidence, subgroup labels, and OOD status;
- tie-aware Spearman rank correlation, confusion by dimension, MAE, ECE, Brier score, ICC(C,k), within-item disagreement range, and subgroup slices;
- mandatory human routing for high disagreement, low confidence, or OOD input;
- paired heldout champion/challenger error regression with a lower 95% confidence bound; approved certificates require both human-alignment and champion/challenger policy passes;
- sealed release-holdout corpus identity/digest, clean contamination status, signed human approval, validity window, revocation state, and exact evidence references.
The contract/Prisma alignment manifest records this evidence as code_only.
Generic model, provider, evaluation, or feedback rows cannot be projected as a
calibrated release certificate.
Blinded runtime judging and semantic diversity#
Creative-autonomy now requires both a ValidatedSemanticDiversity binding and
an approved JudgeCalibrationBinding before constructing a director. The
director verifies that:
- semantic provider/model/version, dimensions, validation corpus/evidence, and validity window are complete and current;
- the certificate is approved, current, non-degraded, human-aligned, bias- probe clean, and champion/challenger passing;
- the certificate's judge IDs exactly equal the runtime pointwise and pairwise panel—neither an uncertified live judge nor an absent certified judge is accepted;
- every release tournament returns blinded forward and reverse identity picks plus evidence references. Single-order verdicts fail loud.
Cycle and review provenance retain the raw pairwise comparison evidence and the gate verdict retains the calibration certificate and panel version.
Production novelty, candidate admission, failed-topic cooldown, and portfolio
clustering use only the evidence-bound semantic model. The semantic model and
version are recorded in the score. Token/trigram and hashing-bag-of-words values
remain available only as explicitly labeled lexical-diagnostic-only features.
BFF composition remains fail-closed until a real semantic binding and real judge
certificate are injected; test certificates and embedders live only in named
test-support files.
V8/V10 diagnostic-only G5#
JudgeVerdict and judgePanel still expose V8's deterministic/LLM refinements
and batch-dependent PCA aggregate for diagnosis and regression analysis. They
can no longer turn release G5 green.
decideRelease now requires a separate CalibratedG5ReleaseEvidence record for
the exact case. It checks verdict/certificate/corpus SHA-256 digests, approved
human authority, heterogeneous-panel state, human-alignment and bias passes,
heldout champion/challenger pass, clean contamination status, human routing,
future dating, expiry, and the final calibrated verdict. Missing evidence blocks
release while naming the diagnostic PCA aggregate without promoting it.
The boundary is threaded through Loom, Daedalus, and V10 Case Files. Loom
accepts governed composition evidence; Daedalus and the V10 cook CLI require an
explicit evidence file; the tomorrow scheduler requires
CASE_FILES_G5_EVIDENCE_FILE. Positive tests inject conspicuously named test
fixtures. Negative tests prove that a 0.99/green heuristic score still blocks
without calibration, and also cover wrong-case, expiry, failed heldout
regression, and missing semantic/judge evidence.
Checklist disposition#
Three repository-enforceable EVAL-003 requirements are complete:
- bag-of-words and V8 PCA/G5 are diagnostic-only for production decisions;
- creative candidate admission/selection uses a validated semantic model and records exact model/version while retaining lexical diagnostics separately;
- model, prompt, rubric, examples, weights, thresholds, provider, and policy changes require approved heldout champion/challenger evidence.
Five items remain open because their completion requires real EVAL-002 human data and observations, not schemas or fixtures:
- operate a real heterogeneous provider/model panel and record any degradation;
- run real blinded human/model comparisons in both orders;
- measure all seven declared bias families on heldout human gold;
- publish the real human-judge correlation, dimensional confusion, calibration, reliability, ICC, disagreement, and subgroup report;
- exercise the high-disagreement, low-confidence, and OOD human route with consented independent raters.
No test fixture is counted as evidence for those five open boxes.
Automated verification#
The final integrated source passed 4,440/4,440 non-duplicated owning-package tests:
| Surface | Result |
|---|---|
| Canonical contracts full package | 4,067/4,067 |
| Creative autonomy full package | 267/267 |
| Affected BFF autonomy route/bindings/E2E/poller | 52/52 |
| Yemaya case-eval full package | 13/13 |
| V8 Loom service full package | 29/29 |
| V10 Case Files cook/scheduler full package | 9/9 |
| V8 Daedalus compiler full package | 3/3 |
The final post-rebase verification repeated V10's 9/9 suite, the 16 focused contract/alignment tests, and the affected contracts, BFF, and case-eval typechecks.
Production typechecks passed for contracts, creative autonomy, case-eval, Loom,
V10 Case Files, Daedalus, and the resource-bounded BFF autonomy graph. The
canonical contracts Nx build and the BFF 29.7 MiB production bundle passed.
Prettier and git diff --check passed. New contracts/test-support files pass
the repository ESLint policy with zero warnings; the remaining modified legacy
files pass ESLint with the repository's existing non-null-assertion warning
disabled and no other warnings or errors.
Representative commands:
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts --testTimeout=120000
pnpm exec vitest run <four affected BFF files> --config apps/oshun/bff/vitest.config.ts --testTimeout=120000
pnpm --dir libs/yemaya/case-eval exec vitest run --testTimeout=120000
pnpm --dir apps/v8/loom-service exec vitest run --testTimeout=120000
pnpm --dir apps/v10/case-files-cook exec vitest run --testTimeout=120000
pnpm --dir apps/v8/daedalus-compiler exec vitest run --testTimeout=120000
pnpm exec tsc -p <each owning package tsconfig> --noEmit
pnpm exec tsc -p apps/oshun/bff/tsconfig.eval003.json --noEmit
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm --dir apps/oshun/bff run build
pnpm exec prettier --check <all changed non-lock files>
git diff --check
Bounded verification notes#
- The first full contracts run passed 4,066 tests and failed only because the
new contract was absent from the exhaustive contract/Prisma alignment table.
After adding the honest
code_onlyentry, the focused suites passed 16/16 and the clean full rerun passed all 176 files and 4,067 tests. - The app-wide BFF typecheck exceeded V8's 4 GiB heap while another Codex held
about 6 GiB and an active Nx daemon held about 0.9 GiB on the 15 GiB RAM/31
GiB swap host. The permanent
tsconfig.eval003.jsonchecks the exact affected BFF autonomy graph; it found a real response-digest property mismatch, which was fixed, and then passed. The BFF production bundle and 52 affected tests also passed. - A case-eval Nx build expanded to 37 dependencies. It stopped after unrelated
existing failures in Iris agents (implicit-any/module resolution) and Isis
client DTS generation (TypeScript 6
baseUrldeprecation). The owning case-eval typecheck/test passed, contracts built, and the remaining app-owned build/typecheck paths passed; no unrelated dependency source was altered. - Loom emits existing missing source-map warnings from shared HTTP-client
generated JavaScript; contracts emit the existing missing
libs/contracts/src/index.js.mapwarning. Neither changes test results. - The source rebase incorporated concurrent Daily Wonder schedule assertions; V10's full suite and all affected typechecks passed afterward.
- The unrelated binary
pnpm-lock.yamlmodification was restored byte-for-byte after the rebase (Git bloba37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks andlibs/urania/were not staged. - No customer UI, PWA, admin, mobile, Unreal, or visual surface changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.
Artifact identity#
| Artifact | SHA-256 |
|---|---|
| Canonical judge-calibration contract | 47692dc759f5ccd4cdeeaa1b7ca1709ce00b7715fa47dfa5e9074b5e09f1e768 |
| Creative director release boundary | 7bf18a1521abb42438bd09891024f50a4aeb0db2ba2454556afe4be40d435290 |
| Validated semantic-diversity boundary | e48103e48525c7e51213aa841ae2ef8bd22733f63c159f48200772924c0d0541 |
| Blinded dual-order tournament record | b2c84bef727c6bd7fc862bba2dd1c511f6e2ed5ee397882f3e59b2c52379f660 |
| V8 calibrated G5 release boundary | 74beb64b72d398a5d28160bc5d69aed9a1f12318598ab2e7abfe2a25713e80bf |
| BFF fail-closed composition | d5823c131a264d029bd171a0f57ee823932fad582b8932f3d320ee3ab422a9d9 |
Publication identity#
- Repository source commit:
47438840440d658eeb7f6b4185fcb1ed68f384fb - Repository source publication tip:
47438840440d658eeb7f6b4185fcb1ed68f384fb - EVAL-003 closure evidence commit: not applicable; EVAL-003 remains open.