# EVAL-003 progress evidence — calibrated model-judge release boundary

> **Status: partially complete, not closed.** The repository now prevents
> lexical diversity and uncalibrated/PCA model scores from making release or
> search decisions. It does not claim that any real human-gold calibration has
> occurred. EVAL-002 still records zero acquired corpus/cohort units.

## Work-package charter

- **Task ID:** `EVAL-003`
- **Owner:** evaluation governance, model-judge calibration, creative-autonomy
  quality, and V8/V10 case release; repository implementation owner: Codex
- **Audit finding:** position-mitigated pairwise judging and several isolated
  calibration statistics existed, but no canonical record bound heterogeneous
  judge identities, human holdouts, bias probes, metrics, routing, approvals,
  and change regression. Creative-autonomy production still selected with
  lexical similarity, while V8's batch-dependent PCA/heuristic G5 verdict was a
  hard release gate.
- **Repository objective:** make uncalibrated judge scores and lexical/PCA
  diversity diagnostic-only; fail closed unless exact, current, human-approved
  calibration and validated semantic evidence are present.
- **External objective:** acquire the EVAL-002 human gold, recruit independent
  raters, run the blinded calibration study and bias probes, and issue the first
  real approved certificates.
- **Rollback strategy:** revert source commit
  `47438840440d658eeb7f6b4185fcb1ed68f384fb`.
- **Evidence-pack location:** this file.

## Canonical judge-calibration evidence

`ModelJudgeCalibrationCertificate` and its supporting records now require:

- exact judge, provider, provider-version, model, model-version, model-family,
  prompt digest/version, rubric version, policy version, and response-privacy
  identity;
- declared provider/model-family minima, computed diversity shortfall, and an
  explicit degraded reason; degraded diversity cannot receive release approval;
- blinded aliases, sealed identity-map digest, both candidate orders, both raw
  order picks, per-order confidence/evidence, and a winner derived from order
  agreement rather than claimed by the caller;
- position, verbosity, style, self-family, reference-leakage, contamination, and
  judge-pleasing bias probes, with exact heldout items and threshold-derived
  pass/fail;
- per-item, per-dimension, per-judge human observations with multiple human
  rating references, confidence, subgroup labels, and OOD status;
- tie-aware Spearman rank correlation, confusion by dimension, MAE, ECE, Brier
  score, ICC(C,k), within-item disagreement range, and subgroup slices;
- mandatory human routing for high disagreement, low confidence, or OOD input;
- paired heldout champion/challenger error regression with a lower 95%
  confidence bound; approved certificates require both human-alignment and
  champion/challenger policy passes;
- sealed release-holdout corpus identity/digest, clean contamination status,
  signed human approval, validity window, revocation state, and exact evidence
  references.

The contract/Prisma alignment manifest records this evidence as `code_only`.
Generic model, provider, evaluation, or feedback rows cannot be projected as a
calibrated release certificate.

## Blinded runtime judging and semantic diversity

Creative-autonomy now requires both a `ValidatedSemanticDiversity` binding and
an approved `JudgeCalibrationBinding` before constructing a director. The
director verifies that:

1. semantic provider/model/version, dimensions, validation corpus/evidence, and
   validity window are complete and current;
2. the certificate is approved, current, non-degraded, human-aligned, bias-
   probe clean, and champion/challenger passing;
3. the certificate's judge IDs exactly equal the runtime pointwise and pairwise
   panel—neither an uncertified live judge nor an absent certified judge is
   accepted;
4. every release tournament returns blinded forward and reverse identity picks
   plus evidence references. Single-order verdicts fail loud.

Cycle and review provenance retain the raw pairwise comparison evidence and the
gate verdict retains the calibration certificate and panel version.

Production novelty, candidate admission, failed-topic cooldown, and portfolio
clustering use only the evidence-bound semantic model. The semantic model and
version are recorded in the score. Token/trigram and hashing-bag-of-words values
remain available only as explicitly labeled `lexical-diagnostic-only` features.
BFF composition remains fail-closed until a real semantic binding and real judge
certificate are injected; test certificates and embedders live only in named
test-support files.

## V8/V10 diagnostic-only G5

`JudgeVerdict` and `judgePanel` still expose V8's deterministic/LLM refinements
and batch-dependent PCA aggregate for diagnosis and regression analysis. They
can no longer turn release G5 green.

`decideRelease` now requires a separate `CalibratedG5ReleaseEvidence` record for
the exact case. It checks verdict/certificate/corpus SHA-256 digests, approved
human authority, heterogeneous-panel state, human-alignment and bias passes,
heldout champion/challenger pass, clean contamination status, human routing,
future dating, expiry, and the final calibrated verdict. Missing evidence blocks
release while naming the diagnostic PCA aggregate without promoting it.

The boundary is threaded through Loom, Daedalus, and V10 Case Files. Loom
accepts governed composition evidence; Daedalus and the V10 cook CLI require an
explicit evidence file; the tomorrow scheduler requires
`CASE_FILES_G5_EVIDENCE_FILE`. Positive tests inject conspicuously named test
fixtures. Negative tests prove that a 0.99/green heuristic score still blocks
without calibration, and also cover wrong-case, expiry, failed heldout
regression, and missing semantic/judge evidence.

## Checklist disposition

Three repository-enforceable EVAL-003 requirements are complete:

- bag-of-words and V8 PCA/G5 are diagnostic-only for production decisions;
- creative candidate admission/selection uses a validated semantic model and
  records exact model/version while retaining lexical diagnostics separately;
- model, prompt, rubric, examples, weights, thresholds, provider, and policy
  changes require approved heldout champion/challenger evidence.

Five items remain open because their completion requires real EVAL-002 human
data and observations, not schemas or fixtures:

- operate a real heterogeneous provider/model panel and record any degradation;
- run real blinded human/model comparisons in both orders;
- measure all seven declared bias families on heldout human gold;
- publish the real human-judge correlation, dimensional confusion, calibration,
  reliability, ICC, disagreement, and subgroup report;
- exercise the high-disagreement, low-confidence, and OOD human route with
  consented independent raters.

No test fixture is counted as evidence for those five open boxes.

## Automated verification

The final integrated source passed **4,440/4,440** non-duplicated owning-package
tests:

| Surface                                         |      Result |
| ----------------------------------------------- | ----------: |
| Canonical contracts full package                | 4,067/4,067 |
| Creative autonomy full package                  |     267/267 |
| Affected BFF autonomy route/bindings/E2E/poller |       52/52 |
| Yemaya case-eval full package                   |       13/13 |
| V8 Loom service full package                    |       29/29 |
| V10 Case Files cook/scheduler full package      |         9/9 |
| V8 Daedalus compiler full package               |         3/3 |

The final post-rebase verification repeated V10's 9/9 suite, the 16 focused
contract/alignment tests, and the affected contracts, BFF, and case-eval
typechecks.

Production typechecks passed for contracts, creative autonomy, case-eval, Loom,
V10 Case Files, Daedalus, and the resource-bounded BFF autonomy graph. The
canonical contracts Nx build and the BFF 29.7 MiB production bundle passed.
Prettier and `git diff --check` passed. New contracts/test-support files pass
the repository ESLint policy with zero warnings; the remaining modified legacy
files pass ESLint with the repository's existing non-null-assertion warning
disabled and no other warnings or errors.

Representative commands:

```text
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts --testTimeout=120000
pnpm exec vitest run <four affected BFF files> --config apps/oshun/bff/vitest.config.ts --testTimeout=120000
pnpm --dir libs/yemaya/case-eval exec vitest run --testTimeout=120000
pnpm --dir apps/v8/loom-service exec vitest run --testTimeout=120000
pnpm --dir apps/v10/case-files-cook exec vitest run --testTimeout=120000
pnpm --dir apps/v8/daedalus-compiler exec vitest run --testTimeout=120000
pnpm exec tsc -p <each owning package tsconfig> --noEmit
pnpm exec tsc -p apps/oshun/bff/tsconfig.eval003.json --noEmit
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm --dir apps/oshun/bff run build
pnpm exec prettier --check <all changed non-lock files>
git diff --check
```

## Bounded verification notes

- The first full contracts run passed 4,066 tests and failed only because the
  new contract was absent from the exhaustive contract/Prisma alignment table.
  After adding the honest `code_only` entry, the focused suites passed 16/16 and
  the clean full rerun passed all 176 files and 4,067 tests.
- The app-wide BFF typecheck exceeded V8's 4 GiB heap while another Codex held
  about 6 GiB and an active Nx daemon held about 0.9 GiB on the 15 GiB RAM/31
  GiB swap host. The permanent `tsconfig.eval003.json` checks the exact affected
  BFF autonomy graph; it found a real response-digest property mismatch, which
  was fixed, and then passed. The BFF production bundle and 52 affected tests
  also passed.
- A case-eval Nx build expanded to 37 dependencies. It stopped after unrelated
  existing failures in Iris agents (implicit-any/module resolution) and Isis
  client DTS generation (TypeScript 6 `baseUrl` deprecation). The owning
  case-eval typecheck/test passed, contracts built, and the remaining app-owned
  build/typecheck paths passed; no unrelated dependency source was altered.
- Loom emits existing missing source-map warnings from shared HTTP-client
  generated JavaScript; contracts emit the existing missing
  `libs/contracts/src/index.js.map` warning. Neither changes test results.
- The source rebase incorporated concurrent Daily Wonder schedule assertions;
  V10's full suite and all affected typechecks passed afterward.
- The unrelated binary `pnpm-lock.yaml` modification was restored byte-for-byte
  after the rebase (Git blob `a37431842e330b3a06b030947c25bebdc72bc2b1`).
  Generated Cargo locks and `libs/urania/` were not staged.
- No customer UI, PWA, admin, mobile, Unreal, or visual surface changed.
  Frontend skill, Playwright, mobile automation, screenshots, and Unreal
  verification were inapplicable.

## Artifact identity

| Artifact                              | SHA-256                                                            |
| ------------------------------------- | ------------------------------------------------------------------ |
| Canonical judge-calibration contract  | `47692dc759f5ccd4cdeeaa1b7ca1709ce00b7715fa47dfa5e9074b5e09f1e768` |
| Creative director release boundary    | `7bf18a1521abb42438bd09891024f50a4aeb0db2ba2454556afe4be40d435290` |
| Validated semantic-diversity boundary | `e48103e48525c7e51213aa841ae2ef8bd22733f63c159f48200772924c0d0541` |
| Blinded dual-order tournament record  | `b2c84bef727c6bd7fc862bba2dd1c511f6e2ed5ee397882f3e59b2c52379f660` |
| V8 calibrated G5 release boundary     | `74beb64b72d398a5d28160bc5d69aed9a1f12318598ab2e7abfe2a25713e80bf` |
| BFF fail-closed composition           | `d5823c131a264d029bd171a0f57ee823932fad582b8932f3d320ee3ab422a9d9` |

## Publication identity

- **Repository source commit:** `47438840440d658eeb7f6b4185fcb1ed68f384fb`
- **Repository source publication tip:**
  `47438840440d658eeb7f6b4185fcb1ed68f384fb`
- **EVAL-003 closure evidence commit:** not applicable; EVAL-003 remains open.
