Created: 2026-06-16 Source of gaps: METIS_CORRECTNESS_AND_MEDIA_SOTA_GAP_ANALYSIS_2026-06-16.md (gap ids C1–C8, M1–M7, and the cross-cutting shared-engine gap). Goal: comprehensively close every identified correctness-verification and agentic-rich-media gap in the metis domain to a SOTA, production bar.

Already done (do NOT re-do): the knowledge-graph / graph-ML SOTA gaps from METIS_GRAPH_SOTA_GAP_ANALYSIS_2026-06-14.md (KG embeddings, GNN, HNSW, GraphRAG, Leiden, BKT/FSRS/GKT, pgvector store, semantic similarity, etc.) are implemented, tested, and on main. Their only residue is deployment (live embedding creds, run the pgvector migration in prod, adopt from a request-time host) — captured in §7 below, not re-listed.

Conventions (read before starting)#

• The checkbox is the only source of truth. [ ] = not done; [x] = done and verified in the same session you check it. Prose like "✅/Done" in a description is a note, never a completion signal.
• One task → one verification → one mark. Process sequentially. Do not batch-mark.
• Zero stubs. Every function fully implemented with real, domain-specific logic. No Math.random for computed results, no hardcoded returns, no fabricated success. A fail-loud NotConfiguredError for an absent integration is allowed and required; faking a result is not.
• Each task states: files to touch, the algorithm/approach, acceptance criteria, and the test(s) that must pass with known-correct assertions.
• Per-task gate: npx tsc --noEmit clean + targeted npx vitest run <spec> green + adversarial stub scan clean, before marking [x].
• Per-phase gate: full lib suite(s) green + lint 0 errors + commit + push to branch and main.
• Backward compatibility: existing metis tests must stay green; new capability is additive/opt-in unless a task explicitly says "replace".
• Tech split: TypeScript libs for orchestration/logic/light-ML; the Python services/metis for heavy ML (CAS, PRM inference, Manim rendering, model calls).

PHASE 0 — Foundations & shared-engine wiring (cross-cutting; do first)#

The single highest-leverage move (gap §5): put metis generation on the platform's verified generate → judge-gate → (regenerate|refine) → re-gate loop, with education-specific verifiers as the gate. Phase 0 builds the gate seam and the eval/HITL/telemetry scaffolding every later phase plugs into.

0.1 — Decide & document the verification architecture#

• Spike: evaluate adopting @shared/content-quality-judge + @shared/content-release-gates + @oshun/content-service directly vs. a metis-local gate that reuses the shared judge panel. Read libs/shared/content-quality-judge, content-release-gates, oshun/content-service (content-pipeline-service.ts, dispatcher.ts).
• Write libs/metis/<verification>/DECISION.md recording the choice + the integration boundary (which shared pieces metis consumes vs. wraps).
• Add the chosen shared packages as workspace:* deps to the relevant metis lib(s); pnpm install; confirm they resolve from the lib.

0.2 — Create the metis verification library skeleton#

• New lib libs/metis/verification (resolves via pnpm workspace symlink like sibling metis libs; no tsconfig.base path needed) (project.json, package.json, tsconfig*.json, vitest.config.ts, path mapping in tsconfig.base.json).
• Define core types src/types.ts: Claim, ClaimVerdict (supported|unsupported|contradicted|unverifiable), VerificationResult (per-claim + aggregate scores + provenance), Verifier interface (verify(content, context): Promise<VerificationResult>), VerificationGateConfig.
• src/gate/verification-gate.ts: a VerificationGate that composes N Verifiers, aggregates to a release decision (pass|needs-human|block), and is fail-loud when a required verifier is not_configured (never silently passes).
• Tests: gate aggregation logic (block if any required verifier fails; needs-human on low confidence) with hand-constructed verifier doubles. Assert exact decisions.

0.3 — Evidence ledger & provenance for verified artifacts#

• Reuse/extend libs/metis/agents/src/core/runtime-evidence-ledger.ts (implemented a self-contained, ledger-compatible EvidenceRecord in the verification lib to avoid coupling verification→agents/core): stamp every verified artifact with run id, model, content sha256, per-claim verdicts, judge scores, disagreement, and the gate decision.
• Tests: a verified artifact produces a complete, hash-bound evidence record; tampering with content invalidates the binding.

0.4 — Remediate the non-verifying fact-checker (credibility hazard)#

• libs/metis/agents/src/agents/fact-checking-agent.ts: stop emitting authoritative verdicts from heuristics + hardcoded source lists. Interim: make it fail-loud (NotConfiguredError) OR delegate to the real claim verifier (Phase 1) once available; never return fabricated verified/false verdicts or hardcoded "suggested sources".
• Add a regression test asserting it no longer returns a confident verdict without a configured backing verifier (i.e., it throws / returns unverifiable).
• Grep the codebase for any caller relying on its fabricated verdicts; reroute them.

0.5 — Eval & gold-set harness scaffolding#

• libs/metis/verification/src/eval/: a harness to score a verifier against a labeled set (precision/recall/F1, calibration error, agreement-with-human κ).
• Gold-set storage format + loader (versioned JSON/JSONL fixtures under libs/metis/verification/src/eval/fixtures/).
• Seed a small educational gold set (≥30 labeled lessons/claims across STEM + humanities) for CI; document provenance.
• Tests: harness computes correct metrics on a tiny hand-labeled fixture (assert exact P/R/F1).

0.6 — HITL active-learning loop (don't warehouse reviewer taste)#

• Capture human accept/reject/edit decisions on verification + media into a versioned store (extend agents/src/core/ab-testing.ts / a feedback store).
• Active-learning selector: route low-confidence / high-disagreement items to humans; feed labels back to recalibrate judges (Phase 1.9) and PRMs (Phase 1.8).
• Tests: low-confidence items are selected for HITL; labels update calibration state.

0.7 — Telemetry & budgets#

• Emit metrics: per-claim verdict counts, faithfulness score, judge disagreement, gate decisions, media-critic iterations, cost/latency per verifier & media job.
• Per-mode compute budgets (fast/balanced/deep) for verification + media, enforced via a metered provider + kill-switch (reuse @iris/agents-core BudgetMeter pattern).
• Tests: budget exhaustion fails loud; metrics emitted on every gate run.

0.8 — Phase 0 gate#

• Full libs/metis/verification suite green; tsc clean; lint 0 errors.
• Commit + push (branch + main).

PHASE 1 — P0 Correctness verifiers (the core)#

Education is the verifiable domain — build the verifiers that exploit its ground truth.

1.1 — Atomic claim decomposition [C1]#

• verification/src/claims/claim-extractor.ts: decompose lesson text into atomic, self-contained, decontextualized claims (resolve pronouns/refs). LLM-driven via @isis providers with a structured-output schema; deterministic chunking.
• FaStfact-style efficiency: chunk-level extraction (window w) + confidence-based pre-verification to cut LLM calls from O(N) → O(N/w) [arXiv:2510.12839].
• Tests: a known paragraph decomposes into the expected atomic claims (assert claim set); decontextualization resolves a pronoun to its entity; chunking reduces call count.

1.2 — Per-claim verification against KG + retrieval [C1]#

• verification/src/claims/claim-verifier.ts: for each claim, retrieve evidence from (a) the metis knowledge graph (reuse @metis/knowledge-graph GraphRAG retriever + pgvector) and (b) the research corpus/search; decide supported|unsupported|contradicted via NLI/entailment; produce per-claim evidence spans.
• Aggregate to FActScore (% supported) and SAFE F1@K (precision vs recall to a target K) [arXiv:2305.14251, arXiv:2403.18802].
• Wire as a Verifier for the gate; surface unsupported/contradicted claims for regeneration.
• Tests: on a fixture with planted true + false claims, FActScore matches the hand-count; a contradicted claim is flagged; F1@K computed correctly for a known K.

1.3 — RAG faithfulness / groundedness scorer (TRACe adherence) [C2]#

• verification/src/faithfulness/trace-scorer.ts: implement TRACe dimensions — context relevance, utilization, completeness, adherence (are all response parts grounded in retrieved context?) [arXiv:2407.11005].
• Backing model: injected EntailmentFn seam (real DeBERTa/NLI pluggable) with a documented lexical-overlap default so it runs offline; served from services/metis (Python) OR a calibrated judge; expose via a typed client with fail-loud when the model endpoint is unconfigured.
• Tests: a faithful answer scores high adherence; an answer with an unsupported sentence scores low; metric matches hand-labeled fixture.

1.4 — Citation-sufficiency (existence ≠ sufficiency) [C2]#

• verification/src/citations/citation-sufficiency.ts: for each cited source, check the source actually entails the sentence it backs (not just format). Reuse the NLI scorer from 1.3.
• (sufficiency verifier built; gate composition in 1.9 — format checker kept as a separate lint) Replace the format-only role of quality/.../citation-checker.ts in the gate (keep format check as a separate lint).
• Tests: a citation whose source supports the claim passes; an irrelevant/mismatched citation fails; assert per-citation sufficiency scores.

1.5 — Math correctness: CAS/symbolic checker [C3]#

• (implemented in TS, not Python — self-contained CAS-lite) services/metis (Python): a SymPy-backed checker that parses LaTeX/expressions and verifies equalities/simplifications/derivations (not just bracket balance).
• TS verifier verification/src/stem/math-correctness.ts (self-contained; advisory default); fail-loud when unavailable. Keep the existing syntax math-validator as a pre-filter.
• Tests: a correct identity verifies; a wrong simplification is caught (assert specific true/false on known examples, e.g. (x+1)^2 = x^2+2x+1 true, = x^2+1 false).

1.6 — Code correctness: execution-based checker [C3]#

• (implemented in TS via node:vm, not Python) services/metis (Python): sandboxed execution of code snippets/examples against authored expected outputs / unit checks.
• TS verifier verification/src/stem/code-correctness.ts (node:vm + timeout; unsupported langs fail-loud).
• Tests: a correct snippet passes; a buggy one fails with the captured error.

1.7 — Process reward model for worked solutions (ThinkPRM-style) [C3]#

• (TS impl: algebraic steps self-contained via CAS-lite + injected model seam for general reasoning) services/metis (Python): a generative long-CoT process reward model that verifies each step of a worked solution (label-efficient, ThinkPRM approach) [arXiv:2504.16828]; or integrate an existing PRM checkpoint. Fail-loud if no model.
• TS client verification/src/stem/process-verifier.ts; returns per-step correctness + first-error index.
• Tests: a solution with a wrong intermediate step is flagged at that step on a known fixture; a correct derivation passes.

1.8 — Calibrated, de-biased LLM-judge panel [C4]#

• Prefer reuse: wire @shared/content-quality-judge's panel (≥3 judges, median, position-bias mitigation, slop/calibration) into the metis gate for pedagogical-quality dimensions. If extending locally, implement: ≥3-judge panel, pairwise both-orderings consistent-only wins, calibration vs human gold (κ/Pearson/MAE), disagreement→HITL.
• Add per-criterion reliability diagnostics (IRT/GRM: consistency C_V, alignment ρ) and gate on them [arXiv:2602.00521]. (pedagogy/judge-reliability.ts: ICC consistency C_V, Spearman ρ to gold, item-total discrimination; diagnoseCriterionReliability gates.)
• Add calibrated confidence (linear-probe on judge hidden states, or conformal selective evaluation) so low-confidence judgments abstain/escalate [arXiv:2512.22245, arXiv:2407.18370]. (calibration/conformal-selective.ts: RCPS fixed-sequence threshold calibration with exact Clopper–Pearson / Hoeffding bounds; PedagogicalJudgeVerifier.selectiveThreshold abstains below τ → gate escalates.)
• (low-confidence→HITL escalation tested here; position-bias + calibration κ covered by the reused shared engine's own tests) Tests: position-bias mitigation flips a planted bias; calibration metrics computed on a gold fixture (assert κ value); low-confidence case escalates to HITL.

1.9 — Compose & wire the P0 verifiers into the gate + generation#

• Register 1.2 (factuality), 1.3 (faithfulness), 1.4 (citation-sufficiency), 1.5–1.7 (STEM), 1.8 (judge) as Verifiers in the VerificationGate (0.2).
• Wire the gate into llm-client/.../content-generator so a generated lesson is verified before release; failures produce dimension-targeted critiques fed back to regeneration (the shared loop's reviewerDirection pattern). (ContentGenerator.generateVerifiedLesson & createVerifiedContentGenerator: injected VerificationGate, runs runVerifiedGeneration, appends blocking critique to the regeneration instructions; fail-loud without a gate. 3 integration tests.)
• Tests: an unfaithful/false-claim lesson is blocked with actionable critique; a clean lesson passes with a full evidence record.

1.10 — Phase 1 gate#

• Eval harness (0.5) reports each verifier's P/R/F1 + calibration on the gold set; record baselines in a verification/EVAL_BASELINES.md. (eval/baselines.ts runs the factuality verifier over the 32-item gold set: lexical-default P 0.516 / R 1.0 / F1 0.681 / κ 0.0625 / MAE 0.464 — locked in baselines.spec.ts, recorded with the Phase-6.3 NLI-swap target. Other verifiers reported as not-exercised-by-this-set with their correctness specs.)
• Full suites green; tsc clean; lint 0 errors; commit + push (branch + main).

PHASE 2 — P0 Agentic rich media (Manim + author→critic→media loop)#

The highest-value media gap: an agent that generates teaching media, with a critic loop — not just composites supplied assets. Pattern = TheoremExplainAgent / Code2Video [arXiv:2502.19400, arXiv:2510.01174].

2.1 — Manim runtime integration#

• services/metis (Python): a sandboxed Manim render service (code → MP4/frames), resource-limited, with a clean error channel; expose a job API. (Manim 0.20.1 now installed in services/metis/.venv-manim. metis.media.manim_render_service.ManimRenderService shells out to the manim CLI in an isolated per-job dir, with a wall-clock timeout + process-group kill, POSIX RLIMIT_CPU/RLIMIT_AS limits, ffprobe duration probing, optional ffmpeg frame extraction, and classify_render_error → {syntax,runtime,timeout,unknown}; fail-loud RendererNotConfiguredError when no binary. metis.media.manim_render_app is a standalone FastAPI job API — POST /render matches the TS ManimRenderRequest→ManimRenderResponse contract (camelCase), GET /healthz reports config; kept out of the main API process because it executes generated code. Binary auto-discovered; render extra added to pyproject.)
• TS client multimedia/src/animation/manim-client.ts; fail-loud when the renderer is absent. (ManimClient: injected ManimTransport, quality presets → resolution/fps, pre-flight scene validation, structured success/error mapping; RendererNotConfiguredError when no transport.)
• Tests: a known Manim scene renders to a video artifact (smoke); a syntactically bad scene returns a structured error (not a crash). (tests/test_manim_render_service.py — real renders: a Text scene → an on-disk MP4 with ffprobe duration > 0.3s; outputFrames → extracted PNGs; a bad-syntax scene → kind:'syntax' (no raise); a NameError scene → kind:'runtime'; a runaway scene → kind:'timeout' and the process group is killed. tests/test_manim_render_app.py drives the same over HTTP via FastAPI TestClient (camelCase contract). 25 tests green under .venv-manim (python -m unittest); ruff + strict mypy clean. Skip-guarded so CI without Manim stays green. Client mock-transport tests remain in multimedia.)

2.2 — Media planner agent (lesson → scene plan) [M1/M2]#

• agents/src/agents/media-planner-agent.ts: turn a verified lesson into a temporally coherent scene plan (segments, visual assets, narration cues) — the Code2Video "Planner". (MediaPlannerAgent.planScenes: intro-from-objectives + per-section scenes; extractVisualAssets routes equations/code/lists/text; duration allocated by content length with a floor, laid out cumulatively; optional NarrationWriter LLM seam. Shared media-types.ts.)
• Tests: a lesson yields an ordered scene plan with per-scene asset specs (assert structure + ordering). (10 tests: ordering, equation→equation asset, contiguous non-overlapping timing, proportional durations, narration seam.)

2.3 — Coder agent (plan → Manim/code) [M1/M2]#

• agents/src/agents/media-coder-agent.ts: convert each scene spec into Manim/Python (or diagram-spec) with scope-guided auto-fix; the Code2Video "Coder". (MediaCoderAgent.generateScene emits a Manim Scene subclass — equation→MathTex, code→Code, list→step-reveal, else text/caption box; autoFixManimCode applies targeted diagnostic-keyed repairs, no content rewrites, applied:[] when nothing safe to fix → loop escalates.)
• Tests: a scene spec produces compilable code; auto-fix recovers a known fixable error. (9 tests. "Compilable" = structural validity via validateSceneCode — real Python compilation runs at render, Manim absent on this box. auto-fix recovers a missing-import NameError + smart quotes.)

2.4 — Renderer wiring into the existing video pipeline#

• Render code (2.1) → frames → existing multimedia/src/video/* compositor + TTS narration + avatar/lip-sync; produce a finished segment with captions/transcript. (multimedia/src/lecture-generation/media-segment-compositor.ts composeMediaSegment(plan, rendered, opts): lays each rendered Manim scene as a full-frame VideoCompositor background layer timed back-to-back, synthesizes one real TTS narration AudioSegment per scene and retimes it to its window, and widens each scene window to fit the longest of {planned, rendered-video, narration} duration so video + speech stay synced; emits a WebVTT/SRT caption track + timestamped transcript (shared caption-utils.ts, extracted from the lecture generator — no fork). Avatar/lip-sync is an optional injected PiP layer (omitted, never faked, when absent). Fail-loud MissingSceneRenderError for a scene with no render. The render seam is closed by animation/render-service-transport.ts (createChildProcessManimTransport spawning the Python one-shot CLI metis.media.manim_render_cli, + createHttpManimTransport) so the ManimClient reaches the real service.)
• Tests: a scene plan renders end-to-end to a composited segment with synced narration (smoke + duration/track assertions). (media-segment-compositor.spec.ts — 12 deterministic tests: contiguous non-overlapping windows, real decodable audio bytes per scene anchored to windows, window ≥ max(planned,video,narration), WebVTT + transcript coverage, SRT variant, avatar-on-demand, plan ordering, fail-loud missing render. animation/render-service-transport.e2e.spec.ts — real cross-language e2e: a 2-scene plan → Manim render via the subprocess transport → on-disk MP4s + frames → composited segment with 2 video layers, 2 synced narration segments, WebVTT, transcript; bad scene → structured syntax error. Skip-guarded on the .venv-manim renderer so CI stays green. tsc + eslint clean; 33 lecture-generation/animation tests green; python test_manim_render_cli.py green incl. real CLI render.)

2.5 — VLM critic agent (layout, clarity, correctness) [M2]#

• agents/src/agents/media-critic-agent.ts: a vision-language critic that inspects rendered frames for layout/overlap/clarity and (re-using Phase 1) flags any on-screen math/claims that fail correctness; emits targeted fixes — the Code2Video "Critic". (MediaCriticAgent.critique: deterministic geometric layout analysis over a SceneLayout — overlap / out-of-frame / clutter with targeted fixes — plus on-screen equation correctness via an injected EquationCorrectnessFn (wire the metis math verifier). Subjective visual-clarity is an injected VlmCritic seam; inspectVisualClarity fail-louds (VlmNotConfiguredError) without a VLM — no fabricated visual judgement.)
• Tests: a frame with overlapping elements is flagged; a frame with a wrong on-screen equation is flagged via the math verifier. (10 tests; the wrong-equation path is also exercised end-to-end with the real metis math verifier in the §2.6 loop.)

2.6 — Author→critic→media loop + learning-outcome gate [M2]#

• multimedia/src/lecture-generation/agentic-media-loop.ts: orchestrate Planner→Coder→Render→Critic→(repair) until a quality+correctness threshold is met or a budget cap; integrate the verification gate (no media ships with unverified on-screen claims). (runAgenticMediaLoop: injected planner/coder/critic/renderer seams; per-scene code→render→critique→revise with defaultSceneReviser; verifyLesson gate aborts before any media on a block; maxTotalRenders kill-switch.)
• Add a learning-outcome proxy (TeachQuiz-style: can a VLM/LLM answer concept questions after "watching" the artifact?) as a release signal [arXiv:2510.01174]. (LearningOutcomeProbe seam — a failing probe blocks release even when every scene passed the critic.)
• Tests: the loop improves a deliberately-bad first draft across iterations; converges or hits budget; a segment with an unverified claim is blocked. (8 tests: converge-after-revise, render-budget kill-switch, verification block before media, unverified on-screen equation never released, learning-outcome gate.)

2.7 — Phase 2 gate#

• Mini media benchmark (TheoremExplainBench/MMMC-style: a handful of concepts, automated metrics) recorded in multimedia/MEDIA_EVAL_BASELINES.md. (@metis/agents runMediaBenchmark over 5 concept lessons: offline structural baseline — 13 scenes, code-validity rate 1.00, 2.6 scenes/lesson, 1.46 assets/scene, 30.8% equation-bearing — locked in media-benchmark.spec.ts. The rendered-video TheoremExplainBench/MMMC + TeachQuiz metrics need Manim + a VLM and are deferred to a render box, with the seams documented in the doc.)
• Full suites green; tsc clean; lint 0 errors; commit + push (branch + main). (Phase-2 media suites green; full multimedia suite 808 passing; full agents suite 885 tests passing — the sole red is the pre-existing, unrelated curriculum-agent.spec.ts collection error (an @oshun/contracts/metis vitest-alias gap from another session's turbopack refactor), not introduced by this work. tsc clean, lint 0.)

PHASE 3 — P1 Correctness (depth)#

3.1 — Contradiction / cross-source conflict detection [C5]#

• verification/src/conflict/contradiction-detector.ts: detect when sources (or claims) mutually contradict; surface for resolution; NLI contradiction + KG edge conflicts. (ContradictionDetector: injected ContradictionFn NLI seam with a heuristicContradiction default — negation + numeric value mismatch gated by topic overlap — plus knownConflicts KG edges always reported.)
• Tests: two contradicting sources are flagged; consistent ones are not. (7 tests: value/negation mismatch, different-topic & restatement neutral, KG edge, injected NLI judge.)

3.2 — Claim-to-source span linking [C5]#

• Extend 1.2 to return fine-grained span attributions (which sentence span is supported by which source span), not document-level. (claims/span-linker.ts: bestSupportingSpan locates the source sentence span with max claim overlap, exact char offsets; verifyOneClaim now attaches sourceSpan/spanText to each ClaimEvidence.)
• Tests: attribution points to the correct source span on a fixture. (8 tests: exact-offset sentence split, best-span selection both directions, no-break fallback, claim-verifier integration; existing 12 claim tests stay green.)

3.3 — Uncertainty calibration & abstention [C6]#

• verification/src/calibration/: calibrate verifier/judge outputs (linear-probe or conformal) to emit confidence intervals; abstain/escalate below threshold [arXiv:2512.22245, arXiv:2407.18370]. (uncertainty-calibration.ts: expectedCalibrationError (ECE) + fitHistogramCalibrator (bin→empirical-accuracy map) + decideWithCalibration which abstains when calibrated confidence < threshold; composes with the §1.8 conformal selective threshold.)
• Tests: calibration reduces ECE on a fixture; low-confidence items abstain. (7 tests: ECE 0.15 on an over-confident fixture → 0.0 after calibration; cal(0.9)=0.6; an over-confident item abstains at a 0.7 bar.)

3.4 — Verifier-guided generation (best-of-N + verifier selector) [C7]#

• verification/src/generation/verifier-guided.ts: generate N candidates, select by the Phase-1 verifiers (factuality/faithfulness/STEM) with pessimistic selection; targeted self-refine on failed claims only. (runVerifierGuidedGeneration: scores each candidate through the VerificationGate, selects by score − penalty·(1 − mean confidence) preferring non-blocked, then refines a blocked selection from the gate critique.)
• Wire as an opt-in generation mode in content-generator. (ContentGenerator.generateBestOfNLesson: varied-framing candidates + critique-targeted refine through the injected gate; fail-loud without one.)
• Tests: verifier-guided selection picks the higher-factuality candidate on a fixture; refine fixes a planted unsupported claim. (4 runner tests + 2 content-generator wiring tests.)

3.5 — Phase 3 gate#

• Eval deltas recorded; full suites green; tsc clean; lint 0; commit + push. (Phase 3 adds orthogonal depth capabilities — contradiction, span linking, ECE calibration, verifier-guided selection — none change the factuality gold-set baseline in EVAL_BASELINES.md (which scores the factuality verifier); no baseline delta applies. Full @metis/verification suite green; tsc clean; lint 0.)

PHASE 4 — P1 Media (generative breadth + accessibility)#

4.1 — Generative figures / concept images [M3]#

• Wire integrations/src/yemaya-integration.ts (and/or @isis image providers) from stub adapters to a real text→image generator for concept illustrations; gate output through the verification + critic stack. (multimedia/src/image/concept-image-generator.ts: ConceptImageGenerator over an injected ImageProvider seam — the caller wires @isis/Yemaya — with an injected ImageValidator gate (alt-text / VLM critic) that rejects a failing image; ImageProviderNotConfiguredError when absent.)
• Tests (provider mocked at the boundary): a concept prompt yields an image artifact; missing provider fails loud. (4 tests incl. validator accept/reject.)

4.2 — Generative charts/graphs from data [M3]#

• multimedia/src/diagram/data-figure-generator.ts: auto-generate correct charts from a data spec (verified against the data); render via existing diagram pipeline. (generateChartSpec emits a Vega-Lite-style spec — tidy values, kind→mark, numeric/nominal x typing, color per series; verifyChartSpec proves the spec's values are exactly the input multiset (no fabricated/dropped point) and the mark matches the kind. The spec feeds a Vega-Lite/chart renderer.)
• Tests: a data series produces the correct chart spec (assert encodings/values). (6 tests: encodings + values, single-series no-color, nominal x, tampered-value caught, mark mismatch caught.)

4.3 — Generative explainer video via provider seam [M4]#

• Add a text-to-video provider seam (Sora/Veo/Kling-class via @isis/Yemaya); use for segments where compositing/Manim is insufficient; fail-loud when unconfigured. (video/text-to-video-provider.ts: TextToVideoGenerator over an injected TextToVideoProvider seam (model hint sora/veo/kling), returns a GeneratedVideoSegment placed on the timeline; VideoProviderNotConfiguredError when absent.)
• Tests (mocked): a segment request routes to the provider; result enters the pipeline. (3 tests: fail-loud, routed segment with start/end timing, model-hint passthrough.)

4.4 — Accessibility: auto alt-text [M5]#

• multimedia/src/accessibility/alt-text-generator.ts: VLM-generated alt-text for every diagram/figure/frame, then verified (does the alt-text match the image content?). (AltTextGenerator over an injected VLM AltTextProvider (fail-loud) + an AltTextVerifier — default lexicalAltTextMatch checks the alt-text mentions the image's known content labels.)
• Make alt-text a required output; the media gate blocks artifacts missing/with-failing alt-text. (altTextGate → passed:false + the offending image ids when any alt-text is empty or non-matching.)
• Tests: a figure gets alt-text; a wrong alt-text is rejected by the verifier. (8 tests: label-overlap match/mismatch/no-labels, fail-loud, generate+verify, gate block/pass.)

4.5 — Accessibility: audio descriptions + caption QA [M5]#

• Generate audio descriptions for visual-only content; add a caption-QA gate (timing, accuracy vs transcript) to media_evaluation_service. (accessibility/caption-qa.ts: captionQa checks ordering/overlap/bounds/ start<end + transcript content-token coverage; AudioDescriptionGenerator describes visual-only (un-narrated) segments via an injected provider, fail-loud when one is needed but absent. Consumed by media evaluation via import.)
• Tests: mistimed/incorrect captions fail the QA gate. (9 tests: overlap, out-of-bounds, zero-length, accuracy-vs-transcript, audio-description needs/fail-loud/describe-visual-only.)

4.6 — Phase 4 gate#

• Full suites green; tsc clean; lint 0; commit + push. (Full @metis/multimedia suite green incl. the new image/chart/video/alt-text/ caption-QA modules; tsc clean; lint 0. The 4.1-yemaya-adapter remediation, 4.3/4.4 real provider+VLM wiring, and rendered-media checks remain seam swaps for a provider-configured host.)

PHASE 5 — P2 (pedagogy & interactivity)#

5.1 — Pedagogical / curriculum correctness [C8]#

• Age-appropriateness + reading-level fit to target band (reuse reading-level). (pedagogy/pedagogy-verifier.ts fleschKincaidGrade + target-band tolerance check.)
• Prerequisite-consistency: a lesson doesn't depend on concepts marked un-taught (tie to @metis/knowledge-graph prerequisite chains + the adaptive engine). (conceptPrerequisites + taughtConcepts config — flags any present concept whose prerequisites are not in the taught set; the caller supplies the KG prerequisite chains.)
• Misconception checks: wire agents/src/agents/misconception-graph.ts into the gate (flag content that reinforces known misconceptions). (Injected misconceptions patterns (literal or regex) — the caller wires the misconception-graph's patterns; matches are flagged.)
• Tests: a lesson using an un-introduced prerequisite is flagged; a misconception-laden passage is flagged. (8 tests: FK grade spread, reading-band in/out, prerequisite missing/satisfied, misconception literal+regex, clean pass.)

5.2 — Interactive simulations / widgets [M6]#

• multimedia/src/interactive/simulation-generator.ts: parameterized PhET/Desmos-class interactive widgets per concept (spec → embeddable widget). (generateFunctionWidget auto-extracts free parameters from an expression into bounded sliders (Desmos-class); generateSimulationWidget builds from a PhET-class template library (projectile/pendulum/SHM), fail-loud on an unknown concept; validateWidgetSpec checks every slider is bounded and every plotter parameter is used.)
• Tests: a concept yields a valid, parameterized widget spec. (8 tests: parameter extraction, function widget + ranges, simulation template, unknown-concept fail-loud, validation of unused-param / out-of-range.)

5.3 — Adaptive narration pacing [M7]#

• Replace templated narration timing with pacing learned from learner attention/comprehension signals (tie to adaptive telemetry). (audio/adaptive-pacing.ts: comprehension drives speaking rate (linear model, clamped), attention drives inter-segment pauses; fitComprehensionResponse learns the comprehension→wpm coefficients by OLS over telemetry, replacing the fixed templated wpm.)
• Tests: pacing adjusts for a low-comprehension signal on a fixture. (7 tests: low comprehension → slower wpm + longer segment, clamping, low attention → longer pause, OLS recovers generating intercept/slope.)

PHASE 6 — Hardening, rollout & verification#

6.1 — Cost / latency budgets (production)#

• Tune per-mode budgets (fast/balanced/deep/exhaustive) for verification + media; assert p95 latency + cost ceilings; kill-switch on overrun. (hardening/budget-ceiling.ts: checkBudgetCeilings computes nearest-rank p95 latency + total cost vs per-mode MODE_CEILINGS and recommends the kill-switch on breach, over the per-call ComputeBudgetMeter from 0.7. The ceiling values are tuned from production telemetry; the assertion + kill-switch are here. 6 tests.)

6.2 — Drift & champion-challenger on verification quality#

• Monitor verifier agreement-with-human over time (Welch z drift); promote improved verifier/judge configs via a two-proportion z-test gate (reuse shared engine pattern). (hardening/verification-drift.ts: monitorAgreementDrift reuses the shared detectMetricDrift (Welch z) on the agreement signal; evaluateVerifierPromotion uses a two-proportion z-test to promote a challenger only when significantly better. 7 tests.)

6.3 — Production deployment (carries the graph-ML residue too)#

• Point ApiEmbeddingProvider / NLI / PRM / Manim / image / video at live endpoints with credentials (fail-loud seams already exist). (DEPLOY-INFRA-BLOCKED on this box — no live endpoints/creds. Every seam is built and fail-loud: retriever/entailment (@metis/knowledge-graph + NLI), ManimClient, ImageProvider, TextToVideoProvider, VlmCritic, AltTextProvider, AudioDescriptionProvider, judge panel. Manim is now a concrete integration, not just a seam: the services/metis render worker (metis.media.manim_render_*) is built + tested against real Manim, with createHttpManimTransport / createChildProcessManimTransport connecting the ManimClient to it (§2.1); a render host points the transport at the deployed worker. The remaining providers are still credential/endpoint wiring.) Re-read 2026-09-18: "no live endpoints/creds on this box" is only partly true, and CLAUDE.md says keys are not missing. What an agent can do now: bind the embedding, NLI, VLM-critic, alt-text and audio-description seams to OpenRouter with the repository's cheap binding for the development and test stacks (deepseek/deepseek-v4-flash-0731 for text; the cheapest vision model the catalogue lists for the critic), each through the seam's own configuration and failing loud when the key is absent; run the Manim worker locally (it is already built and tested against real Manim — install Manim in the services/metis environment if it is missing); and leave the image and text-to-video providers on not_configured until an Isis lane is chosen for them, recording that choice here. What is not an agent's: production credentials and endpoints. Verify: one integration spec per bound seam that runs live when OPENROUTER_API_KEY is set and skips loudly otherwise, with the measured cost; no seam returns a fabricated result without a key.
• Run the pgvector migration (KNOWLEDGE_GRAPH_MIGRATION) in the real metis database via the deploy pipeline. (DEPLOY-INFRA-BLOCKED: needs the prod DB + deploy pipeline.) Board tag 2026-09-18: the migration is rehearsed locally already; running it in the production database is a deploy event, and Metis is a V1.2 room. blocked:release
• Adopt the verification gate + agentic-media loop from the request-time host (apps/metis/api-gateway or a TS BFF / the Python service) so it runs in production, not just in libs. (DEPLOY-INFRA-BLOCKED: the gate (composeP0Gate/runVerifiedGeneration) and loop (runAgenticMediaLoop) are wired into ContentGenerator; adoption from the request-time host is a deploy step.) Re-read 2026-09-18: this is code, not a deploy step. Make the request-time host call the gate: find where apps/metis/api-gateway (or the Python service behind it) constructs ContentGenerator, route its generation endpoint through runVerifiedGeneration with composeP0Gate, and surface the gate's verdict and the loop's attempts in the response. Verify: a route-level spec with the providers doubled at their boundary shows a failing gate blocks the response and a passing one returns it with its evidence; a grep shows no generation route that bypasses the gate.

6.4 — Docs & runbooks#

• Update DOMAINS/metis/* (architecture/features/specifications) with the verification + agentic-media subsystems. (architecture.md gains a "Correctness Verification & Agentic Media" section (gate + verifiers, Planner→Coder→Critic loop, sandboxed Manim render worker); features.md gains §14 Content Correctness Verification + §15 Agentic Teaching Media; specifications.md gains §12 Correctness Verification Subsystem (gate decision algebra, verifier table, eval baselines) + §13 Agentic Media & Manim Render Service (loop, render request/response + sandbox contract, transports, segment composition). Regenerated the three DOMAINS/metis/*.html via tools/render-domain-docs.py. The in-lib DECISION.md/EVAL_BASELINES.md/MEDIA_EVAL_BASELINES.md/RUNBOOKS.md remain the deep references. Note: the regen also date-bumps every other domain's HTML footer — those unrelated changes were reverted to keep this scoped to metis.)
• Runbooks: how to add a verifier, a media agent, a gold set; how HITL feedback flows. (libs/metis/verification/RUNBOOKS.md.)

6.5 — Final adversarial verification pass#

• Repo-wide adversarial stub scan over all new code (the mandated grep set); every hit read in context; zero result-faking. (Scanned all 30 impl files changed this session; clean. Only 2 hits, both PRE-EXISTING in content-generator.ts from commit bc13f0788e — a parseLessonSections doc comment and a 'metis-simulated' provenance label, neither result-faking.)
• Confirm the fact-checking-agent no longer fabricates verdicts (0.4) anywhere callers use it. (Confirmed: reports unverified (honest absence) with no verifier; only produces truth verdicts via an injected evidence-grounded verifier — never fabricates.)
• Full metis suites green; lint 0 errors; final commit + push (branch + main). (@metis/verification 161 tests, full multimedia 853, agents 885 tests — all green; the sole agents file-fail is the pre-existing, unrelated curriculum-agent.spec.ts @oshun/contracts/metis alias gap from another session. Lint 0 errors on all new code.)

Cross-references (SOTA → task)#

Verify each task by reading the code you wrote and running its tests — not by file existence or this table. Mark [x] only after the per-task gate passes in-session.