Status: repository-complete / closed. Canonical, executable batteries now cover every modality and education requirement in EVAL-005, bind evidence to exact artifacts, require the applicable blinded human or controlled study, and fail closed at production release boundaries. This closure does not claim that a live education study or evaluator deployment has occurred; absent real evidence blocks release.
Work-package charter#
- Task ID:
EVAL-005 - Owner: canonical evaluation contracts, V1/V8 multimodal production, and V9 lesson-generation/release maintainers; implementation owner: Elikem Agudogo
- Accountable product: shared V1–V9 autonomous-content evaluation plane, with V9 education as the invoked release integration and MEDIA-001 as the inherited multimodal battery
- Priority: P0-G/P1
- Target milestone: 2026-07-15 SOTA gap-closure program
- Risk tier: high education-efficacy, learner-safety, false-release, and evidence-substitution risk
- Dependencies:
MEDIA-001multimodal batteries; canonical evaluation, semantic-grounding, artifact-rights, signing, and V9 lesson contracts; V9 lesson gates, lesson forge, governance, and Prometheus production pipeline; real qualified-teacher and target-learner study operations for any production release - Affected schemas and content slots: education evaluation schema version 1,
V9 lesson artifact schema
v9.lesson.1, G2 semantic-grounding and G3 teachability/education gates, lesson-forge provider boundary, Prometheus compilation/signature surface, and the V9 education lesson content slot. The image, video, speech/music, and 3D/game slots are inherited from MEDIA-001 and were not changed by the EVAL-005 education source commit. - Linked task IDs:
EVAL-005,MEDIA-001,PROOF-002, andPROOF-003 - Audit finding: MEDIA-001 closed the image, video, audio, 3D, and gameplay battery requirements, but the education battery remained absent and V9's teachability gate did not require exact-artifact outcome evidence.
- Repository objective: complete the education battery, make every required outcome non-compensatory, and bind the result through both V9 generation paths and signed artifact provenance without changing the seven-gate model.
- Rollout strategy: additive strict contracts and explicit test fixtures, followed by fail-closed provider injection and G3 publication enforcement.
- Rollback strategy: revert source commit
5ebde0db9f81c8d3010e9ea0a0185d6cc7311f43; this removes the education contract and restores the prior V9 G3 behavior. - Evidence-pack location: this file.
Closure matrix#
| Ledger requirement | Closure proof |
|---|---|
| Image | MEDIA-001's canonical MediaEvaluationBundle requires composition, prompt/source alignment, identity, typography, aesthetics, cultural/brand correctness, artifact detection, accessibility, and blinded side-randomized preference for the exact artifact. |
| Video | MEDIA-001 requires temporal continuity, direction, edit rhythm, lip sync, story comprehension, flicker/artifacts, captioning, accessibility, and blinded viewer preference. |
| Speech/music | MEDIA-001 requires intelligibility, WER where applicable, speaker/consent match, timing, loudness, silence/artifacts, musical structure, originality/fingerprint safety, accessibility, and blinded expert/listener preference. |
| 3D/game | MEDIA-001 requires topology, UV, rigging, collision, LOD/Nanite, cook, performance, compatibility, novelty, accessibility, playability, fun, player comprehension, and blinded human comparison as applicable. |
| Education | The new canonical EducationEvaluationBundle requires source faithfulness, misconception repair, near transfer, far transfer, delayed retention, calibration, qualified-teacher judgment, learner preference, and a preregistered controlled efficacy study for the exact lesson artifact. |
| External benchmarks | Both media and education contracts retain external benchmarks as diagnostic records only. They cannot satisfy missing probes, human comparisons, controlled-study requirements, or a release threshold. |
The first four rows and the shared benchmark rule inherit their already published source and closure proof from MEDIA-001. EVAL-005 adds the previously missing education implementation and integrates it into V9's actual production and release paths.
Canonical education battery#
EducationEvaluationBundle binds one versioned battery to an exact education
subject and SHA-256 artifact digest. Its eight required dimensions are:
- source faithfulness;
- misconception repair;
- near transfer;
- far transfer;
- delayed retention;
- calibration;
- teacher judgment; and
- learner preference.
Every dimension has an exact population type, score, threshold, derived status,
unique release probe, and evidence references. Claims, learners, and qualified
teachers cannot be substituted for one another. A declared passed status with
a sub-threshold score is rejected instead of trusted.
The controlled target-learner comparison requires a distinct baseline, frozen preregistration, randomized assignment, learner blindness, teacher-review blindness, consent evidence, locales, sample and vote totals, qualified teacher reviewers, and evidence for all eight outcome families. Delayed retention must be measured at least seven days after the intervention. The candidate must win the blinded learner comparison; high machine or benchmark scores cannot compensate for a missing or losing study.
runEducationEvaluationBattery is executable rather than a shape-only schema.
It requires all eight dimension runners and the controlled-study runner, parses
their outputs, and evaluates complete coverage. Missing runners, omitted or
duplicated probes, inflated statuses, wrong subjects/digests, insufficient
retention delay, invalid vote totals, and absent study evidence fail loud or
produce an explicit release hold.
V9 production and release binding#
The existing V9 seven-gate governance model is preserved. G3 remains one gate, but now requires both teachability and a complete education evaluation for the exact assembled lesson artifact. Missing, incomplete, or substituted evidence blocks publication. G2 was also hardened so a declared Sophia source pin must appear in the semantic-grounding citations rather than merely having the right shape.
The lesson forge accepts explicit semantic-grounding and education-evaluation
providers. Neither provider has a production pass-through default: absence fails
closed. Test-only passing fixtures live behind explicit test-support exports
and are not used by production composition.
The Prometheus production pipeline independently constructs the canonical education artifact from ground truth, narrative, media, explorables, assessment, and schedule; derives its canonical subject and digest; obtains evaluation evidence; and supplies that evidence to G3. Artifact compilation retains the evaluation bundle and includes it in the signed surface hash. Publishability requires complete evidence for the exact canonical subject and digest. Provenance verification recomputes the skeleton and surface hashes and verifies the Ed25519 signature, so replacing a study after signing is rejected.
Adversarial fixture matrix#
| Fixture | Result |
|---|---|
| A required dimension runner is absent | Battery execution fails before a partial bundle can authorize release. |
| A required probe is missing or duplicated | Coverage fails; unrelated scores cannot compensate. |
| A caller marks a sub-threshold score as passed | Schema/evaluator rejects the inflated status. |
| Evidence names another subject or digest | Exact-artifact coverage and V9 G3 fail. |
| Retention is measured before seven days | Controlled efficacy study is invalid. |
| Vote totals disagree with the declared sample | Controlled study is invalid. |
| Baseline wins learner preference | Education coverage fails. |
| External benchmark is high but a release probe is absent | Benchmark remains diagnostic; release stays held. |
| V9 has no education provider | Forge/Prometheus fail closed and produce no publishable artifact. |
| Semantic evidence declares but does not cite the source pin | G2 fails. |
| Signed Prometheus artifact receives a substituted study | Recomputed surface hash/signature verification fails. |
Automated verification#
Current admitted refresh#
The current refresh ran against repository commit
a2373f54fa9930fa65cabb0a376d75f42ef8bfa8 with lockfile SHA-256
08b055786e9b8a6104d5313601d7eef3205dedcf395066fd9a8264c9859d690f on host
app-dev-server, Ubuntu 24.04.4 LTS, Linux 6.8.0-134-generic x86_64. The
relevant toolchain was Node.js v24.15.0, pnpm 10.25.0, TypeScript 6.0.3,
and Vitest 4.1.7.
No live model/provider, educator panel, learner cohort, customer data, credential, or deployed V9 service was used. The test-support education bundles are explicitly versioned fixtures; they prove fail-closed composition and exact artifact binding, never a real efficacy outcome.
| UTC interval | Exit | Exact command and result |
|---|---|---|
| 2026-07-29T11:41:57.632Z–2026-07-29T11:41:59.244Z | 0 | pnpm exec vitest run --config libs/contracts/vitest.config.ts libs/contracts/src/common/education-evaluation.test.ts libs/contracts/src/v9/__tests__/v9-contracts.spec.ts libs/contracts/src/common/contract-prisma-alignment.test.ts --maxWorkers=1 — 3 files, 30/30 tests |
| 2026-07-29T11:42:03.998Z–2026-07-29T11:42:08.906Z | 0 | pnpm exec vitest run --config libs/v9/lesson-gates/vitest.config.ts --maxWorkers=1 — 9/9 tests |
| 2026-07-29T11:42:16.071Z–2026-07-29T11:42:23.198Z | 0 | pnpm exec vitest run --config libs/v9/lesson-explorables/vitest.config.ts --maxWorkers=1 — 5 files, 22/22 tests |
| 2026-07-29T11:42:29.488Z–2026-07-29T11:42:34.678Z | 0 | pnpm exec vitest run --config libs/v9/governance/vitest.config.ts --maxWorkers=1 — 10/10 tests |
| 2026-07-29T11:42:42.722Z–2026-07-29T11:42:51.105Z | 0 | pnpm exec vitest run --config libs/v9/prometheus/vitest.config.ts --maxWorkers=1 — 8/8 tests |
| 2026-07-29T11:43:03.040Z–2026-07-29T11:43:17.572Z | 0 | pnpm exec tsc -p libs/contracts/tsconfig.lib.json --noEmit |
| 2026-07-29T11:43:17.574Z–2026-07-29T11:43:28.683Z | 0 | pnpm exec tsc -p libs/v9/lesson-gates/tsconfig.json --noEmit |
| 2026-07-29T11:43:28.684Z–2026-07-29T11:43:40.815Z | 0 | pnpm exec tsc -p libs/v9/lesson-explorables/tsconfig.json --noEmit |
| 2026-07-29T11:43:40.817Z–2026-07-29T11:43:57.780Z | 0 | pnpm exec tsc -p libs/v9/governance/tsconfig.json --noEmit |
| 2026-07-29T11:43:57.781Z–2026-07-29T11:44:15.559Z | 0 | pnpm exec tsc -p libs/v9/prometheus/tsconfig.json --noEmit |
The admitted current suites passed 79/79 value assertions and all five owning typechecks. They cover the applicable boundaries as follows:
- Unit/schema/contract: exact dimension coverage, derived thresholds, population types, artifact digests, study constraints, V9 lesson publishability, and contract-to-persistence alignment.
- Integration: lesson-gate, lesson-forge, seven-gate governance, and complete Prometheus generation/compilation/provenance paths consume the canonical bundle instead of private pass-through shapes.
- Adversarial/negative: missing runners or dimensions, high diagnostic benchmark substitution, wrong artifact bytes, duplicate probes, inflated pass status, early retention measurement, invalid votes, losing learner preference, missing providers, uncited source pins, and signed-study substitution all fail for their intended reasons.
- Repository E2E: the Prometheus pipeline reaches signed artifact compilation and re-verification with explicit injected evaluation evidence. This is not represented as a deployed provider, educator/learner study, or production release.
UI, accessibility automation, screenshot/media probes, mobile/Unreal automation, and production performance/load tests are inapplicable to this evidence-only refresh and the underlying contract/runtime change. No rendered surface, media byte, mobile flow, Unreal asset, or deployed latency boundary was changed. MEDIA-001 retains the separately published media-probe evidence for the inherited image/video/audio/3D/game batteries.
Historical implementation verification#
At source publication, the integrated source passed 4,133/4,133 non-duplicated owning-package tests:
| Surface | Result |
|---|---|
| Canonical contracts full package | 4,085/4,085 |
| V9 lesson gates | 9/9 |
| V9 lesson explorables/forge | 22/22 |
| V9 seven-gate governance | 10/10 |
| V9 Prometheus production pipeline/provenance | 7/7 |
The final exact-source focused reruns passed 30/30 canonical education/V9 contract/alignment tests, 9/9 lesson-gate tests, and 7/7 Prometheus tests. These focused runs overlap the totals above and are not double-counted. MEDIA-001's separately published multimodal implementation passed 4,351/4,351 tests at its closure.
Typechecks passed for contracts, lesson-gates, lesson-explorables, V9
governance, and Prometheus. The canonical contracts emitted TypeScript build and
Nx build passed. Explicit ESLint over every changed production TypeScript file
passed; ignored test files were then checked with --no-ignore and passed with
zero errors or warnings. Prettier and git diff --check passed.
Representative commands:
pnpm --dir libs/contracts exec vitest run --testTimeout=120000
pnpm --dir libs/contracts exec vitest run <education/V9/alignment files>
pnpm --dir libs/v9/lesson-gates exec vitest run --config vitest.config.ts
pnpm --dir libs/v9/lesson-explorables exec vitest run --config vitest.config.ts
pnpm --dir libs/v9/governance exec vitest run --config vitest.config.ts
pnpm --dir libs/v9/prometheus exec vitest run --config vitest.config.ts
pnpm exec tsc --noEmit -p <each owning package tsconfig>
pnpm exec tsc -p libs/contracts/tsconfig.lib.json
NX_DAEMON=false NX_TUI=false pnpm nx run @oshun/contracts:build --parallel=1 --skip-nx-cache
pnpm exec eslint <changed production TypeScript>
pnpm exec eslint --no-ignore <changed test TypeScript>
pnpm exec prettier --check <all changed non-lock files>
git diff --check
Bounded verification notes#
- Before the 2026-07-29 focused refresh, the host had 15 GiB RAM, 31 GiB swap, 11 GiB available memory, and 16 GiB free disk. An unrelated checkout's long-running Vitest worker and this checkout's Nx daemon were active, so no broad contracts/workspace matrix or production build was started; the five small owning suites and five typechecks ran serially.
- The host had 15 GiB RAM and 31 GiB swap with other long-lived agent/Nx processes present. Broad workspace gates were therefore inapplicable under the repository's safety policy; all owning-package checks ran sequentially.
- The full contracts run was supervised and passed 4,085 tests in 178 files. Later canonical digest integration was covered by the 30-test focused rerun, contracts typecheck, and emitting contracts build.
- The source rebase integrated four unrelated V10 commits. No EVAL-005 source file overlapped; the only shared path was the unrelated binary lockfile.
- The pre-existing
pnpm-lock.yamlmodification was preserved byte-for-byte through hooks and rebase (Git bloba37431842e330b3a06b030947c25bebdc72bc2b1). Generated Cargo locks andlibs/urania/were not staged. - The source commit used the hook's documented
SKIP_TYPECHECK=1only after all five owning-package typechecks and the contracts builds had passed. Hook ESLint, Prettier, asset locking, and documentation validation passed. - No customer UI, PWA, admin, mobile, Unreal, or visual behavior changed. Frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.
Artifact identity#
| Artifact | SHA-256 |
|---|---|
| Canonical education evaluation | b379e87e259306579eb0e80c1aa46ab135ed72a3cdf90efe9cc63be7523aaee1 |
| Canonical V9 lesson binding | 0fc16d84017ea55cd5b8b3ff6653dfb31b43ef0f7392103fa9d023b4794a1c50 |
| V9 G2/G3 release gates | 696b0f5ef5503664f810ce77b4e4b8ffae25444fbaf3169d4d38eb8b5d365b23 |
| V9 fail-closed lesson forge | 51857787a41c933f915cb97393f9bc753dfc7eb941adf726b955572cb571002c |
| Prometheus education pipeline | 52d20e2dc066880532ecabf6405cd15985c7b5e1da3429e14dfe661a6b5e134a |
| Prometheus release gates | db94ae3639b4d1754fa20df76c04d1e13800d15f6628642a5f22f9e7e53a1be7 |
| Prometheus signed compilation | 7e9dd3bffaa2a2b8cbd3afe5b98303a7d7cd67790ba121659c3c3d741a19c654 |
These hashes identify current source at the admitted refresh commit. Later V9
privacy and retrieval-proof work invalidated the original Prometheus pipeline
and compilation hashes; the current 79 tests and five typechecks cover the
integrated source. Governing versions are education evaluation schema 1 and V9
lesson schema v9.lesson.1. Test fixtures use probe/methodology/rater/policy
references ending in v1 or 2026.07.1; those are test identities, not
production rubric, policy, human-study, provider, or model versions.
Release, human-authority, blocker, and generated-status disposition#
- The repository verdict is
repository-complete / closedbecause the contracts and invoked V9 boundaries exist and fail closed. Production release remainsblockedunless the exact lesson carries real probe evidence and a valid preregistered controlled study from the target learner population. - The contract requires qualified teachers, consent evidence, learner and teacher blindness, and target-learner outcomes for release. No such human decision is needed to publish the repository contract itself, and no reviewer identity, signature, teacher judgment, learner preference, or efficacy result is fabricated in this pack.
- Remaining external dependencies are real scorer services, a frozen preregistration, consented target learners, qualified teachers, delayed retention collection, provider/runtime deployment, and exact-artifact release evidence. Fixtures and high external benchmark scores cannot discharge them.
- No generated capability/status artifact claims live educational efficacy or a production release. The completed-work-package admission verifier derives only this checked task's evidence-pack presence and identity; no generated green outcome was hand-edited for this refresh.
Scope and remaining seams#
- Real education dimension scorers, controlled-study operations, qualified teacher panels, and learner cohorts remain deployment services behind the canonical runner/provider interfaces. Their absence blocks publication; this package does not present fixtures as human evidence.
- The battery establishes release authority and exact-artifact integrity. It does not claim population superiority, a completed field trial, or a live provider deployment.
- External benchmark records remain useful for diagnosis and regression triage, but never grant release authority.
Publication identity#
- MEDIA-001 source commit:
f8046ba3a573f082aa24e95573cd30de8a5c31d6 - MEDIA-001 closure evidence commit:
573cce758a2d33cb5caaee8c0e824da09a0eb966 - EVAL-005 source commit:
5ebde0db9f81c8d3010e9ea0a0185d6cc7311f43 - EVAL-005 source publication tip:
5ebde0db9f81c8d3010e9ea0a0185d6cc7311f43 - EVAL-005 closure evidence commit:
9a1e240288e9aeb7778c205faf1b172c004191e9 - Current refresh source commit:
a2373f54fa9930fa65cabb0a376d75f42ef8bfa8 - Current refresh evidence commit: this evidence-only publication commit; the immutable identity is the Git commit containing this file.