# Metis Gap-Closure TODOs — Correctness Verification & Agentic Teaching Media

**Created:** 2026-06-16 **Source of gaps:**
`METIS_CORRECTNESS_AND_MEDIA_SOTA_GAP_ANALYSIS_2026-06-16.md` (gap ids C1–C8,
M1–M7, and the cross-cutting shared-engine gap). **Goal:** comprehensively close
every identified correctness-verification and agentic-rich-media gap in the
`metis` domain to a SOTA, production bar.

> **Already done (do NOT re-do):** the knowledge-graph / graph-ML SOTA gaps from
> `METIS_GRAPH_SOTA_GAP_ANALYSIS_2026-06-14.md` (KG embeddings, GNN, HNSW,
> GraphRAG, Leiden, BKT/FSRS/GKT, pgvector store, semantic similarity, etc.) are
> **implemented, tested, and on `main`**. Their only residue is deployment (live
> embedding creds, run the pgvector migration in prod, adopt from a request-time
> host) — captured in §7 below, not re-listed.

## Conventions (read before starting)

- **The checkbox is the only source of truth.** `[ ]` = not done; `[x]` = done
  **and verified in the same session you check it**. Prose like "✅/Done" in a
  description is a note, never a completion signal.
- **One task → one verification → one mark.** Process sequentially. Do not
  batch-mark.
- **Zero stubs.** Every function fully implemented with real, domain-specific
  logic. No `Math.random` for computed results, no hardcoded returns, no
  fabricated success. A fail-loud `NotConfiguredError` for an absent integration
  is allowed and required; faking a result is not.
- **Each task states: files to touch, the algorithm/approach, acceptance
  criteria, and the test(s) that must pass with known-correct assertions.**
- **Per-task gate:** `npx tsc --noEmit` clean + targeted `npx vitest run <spec>`
  green + adversarial stub scan clean, before marking `[x]`.
- **Per-phase gate:** full lib suite(s) green + lint 0 errors + commit + push to
  branch and `main`.
- **Backward compatibility:** existing metis tests must stay green; new
  capability is additive/opt-in unless a task explicitly says "replace".
- **Tech split:** TypeScript libs for orchestration/logic/light-ML; the Python
  `services/metis` for heavy ML (CAS, PRM inference, Manim rendering, model
  calls).

---

# PHASE 0 — Foundations & shared-engine wiring (cross-cutting; do first)

The single highest-leverage move (gap §5): put metis generation on the
platform's verified `generate → judge-gate → (regenerate|refine) → re-gate`
loop, with education-specific verifiers as the gate. Phase 0 builds the gate
seam and the eval/HITL/telemetry scaffolding every later phase plugs into.

### 0.1 — Decide & document the verification architecture

- [x] Spike: evaluate adopting `@shared/content-quality-judge` +
      `@shared/content-release-gates` + `@oshun/content-service` directly vs. a
      metis-local gate that _reuses_ the shared judge panel. Read
      `libs/shared/content-quality-judge`, `content-release-gates`,
      `oshun/content-service` (`content-pipeline-service.ts`, `dispatcher.ts`).
- [x] Write `libs/metis/<verification>/DECISION.md` recording the choice + the
      integration boundary (which shared pieces metis consumes vs. wraps).
- [x] Add the chosen shared packages as `workspace:*` deps to the relevant metis
      lib(s); `pnpm install`; confirm they resolve from the lib.

### 0.2 — Create the metis verification library skeleton

- [x] New lib `libs/metis/verification` (resolves via pnpm workspace symlink
      like sibling metis libs; no tsconfig.base path needed) (`project.json`,
      `package.json`, `tsconfig*.json`, `vitest.config.ts`, path mapping in
      `tsconfig.base.json`).
- [x] Define core types `src/types.ts`: `Claim`, `ClaimVerdict`
      (`supported|unsupported|contradicted|unverifiable`), `VerificationResult`
      (per-claim + aggregate scores + provenance), `Verifier` interface
      (`verify(content, context): Promise<VerificationResult>`),
      `VerificationGateConfig`.
- [x] `src/gate/verification-gate.ts`: a `VerificationGate` that composes N
      `Verifier`s, aggregates to a release decision (`pass|needs-human|block`),
      and is **fail-loud** when a required verifier is `not_configured` (never
      silently passes).
- [x] Tests: gate aggregation logic (block if any required verifier fails;
      needs-human on low confidence) with hand-constructed verifier doubles.
      Assert exact decisions.

### 0.3 — Evidence ledger & provenance for verified artifacts

- [x] Reuse/extend `libs/metis/agents/src/core/runtime-evidence-ledger.ts`
      (implemented a self-contained, ledger-compatible `EvidenceRecord` in the
      verification lib to avoid coupling verification→agents/core): stamp every
      verified artifact with run id, model, content sha256, per-claim verdicts,
      judge scores, disagreement, and the gate decision.
- [x] Tests: a verified artifact produces a complete, hash-bound evidence
      record; tampering with content invalidates the binding.

### 0.4 — Remediate the non-verifying fact-checker (credibility hazard)

- [x] `libs/metis/agents/src/agents/fact-checking-agent.ts`: stop emitting
      authoritative verdicts from heuristics + hardcoded source lists. Interim:
      make it **fail-loud** (`NotConfiguredError`) OR delegate to the real claim
      verifier (Phase 1) once available; never return fabricated
      `verified/false` verdicts or hardcoded "suggested sources".
- [x] Add a regression test asserting it no longer returns a confident verdict
      without a configured backing verifier (i.e., it throws / returns
      `unverifiable`).
- [x] Grep the codebase for any caller relying on its fabricated verdicts;
      reroute them.

### 0.5 — Eval & gold-set harness scaffolding

- [x] `libs/metis/verification/src/eval/`: a harness to score a verifier against
      a labeled set (precision/recall/F1, calibration error,
      agreement-with-human κ).
- [x] Gold-set storage format + loader (versioned JSON/JSONL fixtures under
      `libs/metis/verification/src/eval/fixtures/`).
- [x] Seed a small educational gold set (≥30 labeled lessons/claims across
      STEM + humanities) for CI; document provenance.
- [x] Tests: harness computes correct metrics on a tiny hand-labeled fixture
      (assert exact P/R/F1).

### 0.6 — HITL active-learning loop (don't warehouse reviewer taste)

- [x] Capture human accept/reject/edit decisions on verification + media into a
      versioned store (extend `agents/src/core/ab-testing.ts` / a feedback
      store).
- [x] Active-learning selector: route low-confidence / high-disagreement items
      to humans; feed labels back to recalibrate judges (Phase 1.9) and PRMs
      (Phase 1.8).
- [x] Tests: low-confidence items are selected for HITL; labels update
      calibration state.

### 0.7 — Telemetry & budgets

- [x] Emit metrics: per-claim verdict counts, faithfulness score, judge
      disagreement, gate decisions, media-critic iterations, cost/latency per
      verifier & media job.
- [x] Per-mode compute budgets (fast/balanced/deep) for verification + media,
      enforced via a metered provider + kill-switch (reuse `@iris/agents-core`
      BudgetMeter pattern).
- [x] Tests: budget exhaustion fails loud; metrics emitted on every gate run.

### 0.8 — Phase 0 gate

- [x] Full `libs/metis/verification` suite green; tsc clean; lint 0 errors.
- [x] Commit + push (branch + main).

---

# PHASE 1 — P0 Correctness verifiers (the core)

Education is the verifiable domain — build the verifiers that exploit its ground
truth.

### 1.1 — Atomic claim decomposition [C1]

- [x] `verification/src/claims/claim-extractor.ts`: decompose lesson text into
      atomic, self-contained, decontextualized claims (resolve pronouns/refs).
      LLM-driven via `@isis` providers with a structured-output schema;
      deterministic chunking.
- [x] FaStfact-style efficiency: **chunk-level** extraction (window `w`) +
      confidence-based pre-verification to cut LLM calls from O(N) → O(N/w)
      [arXiv:2510.12839].
- [x] Tests: a known paragraph decomposes into the expected atomic claims
      (assert claim set); decontextualization resolves a pronoun to its entity;
      chunking reduces call count.

### 1.2 — Per-claim verification against KG + retrieval [C1]

- [x] `verification/src/claims/claim-verifier.ts`: for each claim, retrieve
      evidence from (a) the metis knowledge graph (reuse
      `@metis/knowledge-graph` GraphRAG retriever + pgvector) and (b) the
      research corpus/search; decide `supported|unsupported|contradicted` via
      NLI/entailment; produce per-claim evidence spans.
- [x] Aggregate to **FActScore** (% supported) and **SAFE F1@K** (precision vs
      recall to a target K) [arXiv:2305.14251, arXiv:2403.18802].
- [x] Wire as a `Verifier` for the gate; surface unsupported/contradicted claims
      for regeneration.
- [x] Tests: on a fixture with planted true + false claims, FActScore matches
      the hand-count; a contradicted claim is flagged; F1@K computed correctly
      for a known K.

### 1.3 — RAG faithfulness / groundedness scorer (TRACe adherence) [C2]

- [x] `verification/src/faithfulness/trace-scorer.ts`: implement TRACe
      dimensions — context **relevance**, **utilization**, **completeness**,
      **adherence** (are _all_ response parts grounded in retrieved context?)
      [arXiv:2407.11005].
- [x] Backing model: injected `EntailmentFn` seam (real DeBERTa/NLI pluggable)
      with a documented lexical-overlap default so it runs offline; ~~served
      from `services/metis`~~ (Python) OR a calibrated judge; expose via a typed
      client with fail-loud when the model endpoint is unconfigured.
- [x] Tests: a faithful answer scores high adherence; an answer with an
      unsupported sentence scores low; metric matches hand-labeled fixture.

### 1.4 — Citation-sufficiency (existence ≠ sufficiency) [C2]

- [x] `verification/src/citations/citation-sufficiency.ts`: for each cited
      source, check the source actually **entails** the sentence it backs (not
      just format). Reuse the NLI scorer from 1.3.
- [x] (sufficiency verifier built; gate composition in 1.9 — format checker kept
      as a separate lint) Replace the format-only role of
      `quality/.../citation-checker.ts` in the gate (keep format check as a
      separate lint).
- [x] Tests: a citation whose source supports the claim passes; an
      irrelevant/mismatched citation fails; assert per-citation sufficiency
      scores.

### 1.5 — Math correctness: CAS/symbolic checker [C3]

- [x] (implemented in TS, not Python — self-contained CAS-lite) `services/metis`
      (Python): a SymPy-backed checker that parses LaTeX/expressions and
      verifies equalities/simplifications/derivations (not just bracket
      balance).
- [x] TS verifier `verification/src/stem/math-correctness.ts` (self-contained;
      advisory default); fail-loud when unavailable. Keep the existing syntax
      `math-validator` as a pre-filter.
- [x] Tests: a correct identity verifies; a wrong simplification is caught
      (assert specific true/false on known examples, e.g. `(x+1)^2 = x^2+2x+1`
      true, `= x^2+1` false).

### 1.6 — Code correctness: execution-based checker [C3]

- [x] (implemented in TS via node:vm, not Python) `services/metis` (Python):
      sandboxed execution of code snippets/examples against authored expected
      outputs / unit checks.
- [x] TS verifier `verification/src/stem/code-correctness.ts` (node:vm +
      timeout; unsupported langs fail-loud).
- [x] Tests: a correct snippet passes; a buggy one fails with the captured
      error.

### 1.7 — Process reward model for worked solutions (ThinkPRM-style) [C3]

- [x] (TS impl: algebraic steps self-contained via CAS-lite + injected model
      seam for general reasoning) `services/metis` (Python): a generative
      long-CoT **process reward model** that verifies each _step_ of a worked
      solution (label-efficient, ThinkPRM approach) [arXiv:2504.16828]; or
      integrate an existing PRM checkpoint. Fail-loud if no model.
- [x] TS client `verification/src/stem/process-verifier.ts`; returns per-step
      correctness + first-error index.
- [x] Tests: a solution with a wrong intermediate step is flagged at that step
      on a known fixture; a correct derivation passes.

### 1.8 — Calibrated, de-biased LLM-judge panel [C4]

- [x] Prefer reuse: wire `@shared/content-quality-judge`'s panel (≥3 judges,
      median, position-bias mitigation, slop/calibration) into the metis gate
      for pedagogical-quality dimensions. If extending locally, implement:
      ≥3-judge panel, pairwise both-orderings consistent-only wins, calibration
      vs human gold (κ/Pearson/MAE), disagreement→HITL.
- [x] Add **per-criterion reliability diagnostics** (IRT/GRM: consistency C_V,
      alignment ρ) and gate on them [arXiv:2602.00521].
      (`pedagogy/judge-reliability.ts`: ICC consistency C_V, Spearman ρ to gold,
      item-total discrimination; `diagnoseCriterionReliability` gates.)
- [x] Add **calibrated confidence** (linear-probe on judge hidden states, or
      conformal selective evaluation) so low-confidence judgments
      abstain/escalate [arXiv:2512.22245, arXiv:2407.18370].
      (`calibration/conformal-selective.ts`: RCPS fixed-sequence threshold
      calibration with exact Clopper–Pearson / Hoeffding bounds;
      `PedagogicalJudgeVerifier.selectiveThreshold` abstains below τ → gate
      escalates.)
- [x] (low-confidence→HITL escalation tested here; position-bias + calibration κ
      covered by the reused shared engine's own tests) Tests: position-bias
      mitigation flips a planted bias; calibration metrics computed on a gold
      fixture (assert κ value); low-confidence case escalates to HITL.

### 1.9 — Compose & wire the P0 verifiers into the gate + generation

- [x] Register 1.2 (factuality), 1.3 (faithfulness), 1.4 (citation-sufficiency),
      1.5–1.7 (STEM), 1.8 (judge) as `Verifier`s in the `VerificationGate`
      (0.2).
- [x] Wire the gate into `llm-client/.../content-generator` so a generated
      lesson is verified before release; failures produce dimension-targeted
      critiques fed back to regeneration (the shared loop's `reviewerDirection`
      pattern). (`ContentGenerator.generateVerifiedLesson` &
      `createVerifiedContentGenerator`: injected `VerificationGate`, runs
      `runVerifiedGeneration`, appends blocking critique to the regeneration
      instructions; fail-loud without a gate. 3 integration tests.)
- [x] Tests: an unfaithful/false-claim lesson is blocked with actionable
      critique; a clean lesson passes with a full evidence record.

### 1.10 — Phase 1 gate

- [x] Eval harness (0.5) reports each verifier's P/R/F1 + calibration on the
      gold set; record baselines in a `verification/EVAL_BASELINES.md`.
      (`eval/baselines.ts` runs the factuality verifier over the 32-item gold
      set: lexical-default P 0.516 / R 1.0 / F1 0.681 / κ 0.0625 / MAE 0.464 —
      locked in `baselines.spec.ts`, recorded with the Phase-6.3 NLI-swap
      target. Other verifiers reported as not-exercised-by-this-set with their
      correctness specs.)
- [x] Full suites green; tsc clean; lint 0 errors; commit + push (branch +
      main).

---

# PHASE 2 — P0 Agentic rich media (Manim + author→critic→media loop)

The highest-value media gap: an agent that _generates_ teaching media, with a
critic loop — not just composites supplied assets. Pattern = TheoremExplainAgent
/ Code2Video [arXiv:2502.19400, arXiv:2510.01174].

### 2.1 — Manim runtime integration

- [x] `services/metis` (Python): a sandboxed Manim render service (code →
      MP4/frames), resource-limited, with a clean error channel; expose a job
      API. (Manim 0.20.1 now installed in `services/metis/.venv-manim`.
      `metis.media.manim_render_service.ManimRenderService` shells out to the
      `manim` CLI in an isolated per-job dir, with a wall-clock timeout +
      process-group kill, POSIX `RLIMIT_CPU`/`RLIMIT_AS` limits, ffprobe
      duration probing, optional ffmpeg frame extraction, and
      `classify_render_error` → {syntax,runtime,timeout,unknown}; fail-loud
      `RendererNotConfiguredError` when no binary.
      `metis.media.manim_render_app` is a **standalone** FastAPI job API —
      `POST /render` matches the TS `ManimRenderRequest`→`ManimRenderResponse`
      contract (camelCase), `GET /healthz` reports config; kept out of the main
      API process because it executes generated code. Binary auto-discovered;
      `render` extra added to pyproject.)
- [x] TS client `multimedia/src/animation/manim-client.ts`; fail-loud when the
      renderer is absent. (`ManimClient`: injected `ManimTransport`, quality
      presets → resolution/fps, pre-flight scene validation, structured
      success/error mapping; `RendererNotConfiguredError` when no transport.)
- [x] Tests: a known Manim scene renders to a video artifact (smoke); a
      syntactically bad scene returns a structured error (not a crash).
      (`tests/test_manim_render_service.py` — **real renders**: a `Text` scene →
      an on-disk MP4 with ffprobe duration > 0.3s; `outputFrames` → extracted
      PNGs; a bad-syntax scene → `kind:'syntax'` (no raise); a `NameError` scene
      → `kind:'runtime'`; a runaway scene → `kind:'timeout'` and the process
      group is killed. `tests/test_manim_render_app.py` drives the same over
      HTTP via FastAPI `TestClient` (camelCase contract). 25 tests green under
      `.venv-manim` (`python -m unittest`); ruff + strict mypy clean.
      Skip-guarded so CI without Manim stays green. Client mock-transport tests
      remain in multimedia.)

### 2.2 — Media planner agent (lesson → scene plan) [M1/M2]

- [x] `agents/src/agents/media-planner-agent.ts`: turn a verified lesson into a
      temporally coherent scene plan (segments, visual assets, narration cues) —
      the Code2Video "Planner". (`MediaPlannerAgent.planScenes`:
      intro-from-objectives + per-section scenes; `extractVisualAssets` routes
      equations/code/lists/text; duration allocated by content length with a
      floor, laid out cumulatively; optional `NarrationWriter` LLM seam. Shared
      `media-types.ts`.)
- [x] Tests: a lesson yields an ordered scene plan with per-scene asset specs
      (assert structure + ordering). (10 tests: ordering, equation→equation
      asset, contiguous non-overlapping timing, proportional durations,
      narration seam.)

### 2.3 — Coder agent (plan → Manim/code) [M1/M2]

- [x] `agents/src/agents/media-coder-agent.ts`: convert each scene spec into
      Manim/Python (or diagram-spec) with scope-guided auto-fix; the Code2Video
      "Coder". (`MediaCoderAgent.generateScene` emits a Manim `Scene` subclass —
      equation→`MathTex`, code→`Code`, list→step-reveal, else text/caption box;
      `autoFixManimCode` applies targeted diagnostic-keyed repairs, no content
      rewrites, `applied:[]` when nothing safe to fix → loop escalates.)
- [x] Tests: a scene spec produces compilable code; auto-fix recovers a known
      fixable error. (9 tests. "Compilable" = structural validity via
      `validateSceneCode` — real Python compilation runs at render, Manim absent
      on this box. auto-fix recovers a missing-import NameError + smart quotes.)

### 2.4 — Renderer wiring into the existing video pipeline

- [x] Render code (2.1) → frames → existing `multimedia/src/video/*`
      compositor + TTS narration + avatar/lip-sync; produce a finished segment
      with captions/transcript.
      (`multimedia/src/lecture-generation/media-segment-compositor.ts`
      `composeMediaSegment(plan, rendered, opts)`: lays each rendered Manim
      scene as a full-frame `VideoCompositor` background layer timed
      back-to-back, synthesizes one **real** TTS narration `AudioSegment` per
      scene and retimes it to its window, and widens each scene window to fit
      the longest of {planned, rendered-video, narration} duration so video +
      speech stay **synced**; emits a WebVTT/SRT caption track + timestamped
      transcript (shared `caption-utils.ts`, extracted from the lecture
      generator — no fork). Avatar/lip-sync is an optional injected PiP layer
      (omitted, never faked, when absent). Fail-loud `MissingSceneRenderError`
      for a scene with no render. The render seam is closed by
      `animation/render-service-transport.ts`
      (`createChildProcessManimTransport` spawning the Python one-shot CLI
      `metis.media.manim_render_cli`, + `createHttpManimTransport`) so the
      `ManimClient` reaches the real service.)
- [x] Tests: a scene plan renders end-to-end to a composited segment with synced
      narration (smoke + duration/track assertions).
      (`media-segment-compositor.spec.ts` — 12 deterministic tests: contiguous
      non-overlapping windows, real decodable audio bytes per scene anchored to
      windows, window ≥ max(planned,video,narration), WebVTT + transcript
      coverage, SRT variant, avatar-on-demand, plan ordering, fail-loud missing
      render. `animation/render-service-transport.e2e.spec.ts` — **real
      cross-language e2e**: a 2-scene plan → Manim render via the subprocess
      transport → on-disk MP4s + frames → composited segment with 2 video
      layers, 2 synced narration segments, WebVTT, transcript; bad scene →
      structured `syntax` error. Skip-guarded on the `.venv-manim` renderer so
      CI stays green. tsc + eslint clean; 33 lecture-generation/animation tests
      green; python `test_manim_render_cli.py` green incl. real CLI render.)

### 2.5 — VLM critic agent (layout, clarity, correctness) [M2]

- [x] `agents/src/agents/media-critic-agent.ts`: a vision-language critic that
      inspects rendered frames for layout/overlap/clarity and (re-using Phase 1)
      flags any on-screen math/claims that fail correctness; emits targeted
      fixes — the Code2Video "Critic". (`MediaCriticAgent.critique`:
      deterministic geometric layout analysis over a `SceneLayout` — overlap /
      out-of-frame / clutter with targeted fixes — plus on-screen equation
      correctness via an injected `EquationCorrectnessFn` (wire the metis math
      verifier). Subjective visual-clarity is an injected `VlmCritic` seam;
      `inspectVisualClarity` fail-louds (`VlmNotConfiguredError`) without a VLM
      — no fabricated visual judgement.)
- [x] Tests: a frame with overlapping elements is flagged; a frame with a wrong
      on-screen equation is flagged via the math verifier. (10 tests; the
      wrong-equation path is also exercised end-to-end with the real metis math
      verifier in the §2.6 loop.)

### 2.6 — Author→critic→media loop + learning-outcome gate [M2]

- [x] `multimedia/src/lecture-generation/agentic-media-loop.ts`: orchestrate
      Planner→Coder→Render→Critic→(repair) until a quality+correctness threshold
      is met or a budget cap; integrate the verification gate (no media ships
      with unverified on-screen claims). (`runAgenticMediaLoop`: injected
      planner/coder/critic/renderer seams; per-scene code→render→critique→revise
      with `defaultSceneReviser`; `verifyLesson` gate aborts before any media on
      a `block`; `maxTotalRenders` kill-switch.)
- [x] Add a **learning-outcome proxy** (TeachQuiz-style: can a VLM/LLM answer
      concept questions after "watching" the artifact?) as a release signal
      [arXiv:2510.01174]. (`LearningOutcomeProbe` seam — a failing probe blocks
      release even when every scene passed the critic.)
- [x] Tests: the loop improves a deliberately-bad first draft across iterations;
      converges or hits budget; a segment with an unverified claim is blocked.
      (8 tests: converge-after-revise, render-budget kill-switch, verification
      block before media, unverified on-screen equation never released,
      learning-outcome gate.)

### 2.7 — Phase 2 gate

- [x] Mini media benchmark (TheoremExplainBench/MMMC-style: a handful of
      concepts, automated metrics) recorded in
      `multimedia/MEDIA_EVAL_BASELINES.md`. (`@metis/agents` `runMediaBenchmark`
      over 5 concept lessons: **offline structural** baseline — 13 scenes,
      **code-validity rate 1.00**, 2.6 scenes/lesson, 1.46 assets/scene, 30.8%
      equation-bearing — locked in `media-benchmark.spec.ts`. The rendered-video
      TheoremExplainBench/MMMC + TeachQuiz metrics need Manim + a VLM and are
      deferred to a render box, with the seams documented in the doc.)
- [x] Full suites green; tsc clean; lint 0 errors; commit + push (branch +
      main). (Phase-2 media suites green; full multimedia suite 808 passing;
      full agents suite 885 tests passing — the **sole** red is the
      pre-existing, unrelated `curriculum-agent.spec.ts` collection error (an
      `@oshun/contracts/metis` vitest-alias gap from another session's turbopack
      refactor), not introduced by this work. tsc clean, lint 0.)

---

# PHASE 3 — P1 Correctness (depth)

### 3.1 — Contradiction / cross-source conflict detection [C5]

- [x] `verification/src/conflict/contradiction-detector.ts`: detect when sources
      (or claims) mutually contradict; surface for resolution; NLI
      contradiction + KG edge `conflicts`. (`ContradictionDetector`: injected
      `ContradictionFn` NLI seam with a `heuristicContradiction` default —
      negation + numeric value mismatch gated by topic overlap — plus
      `knownConflicts` KG edges always reported.)
- [x] Tests: two contradicting sources are flagged; consistent ones are not. (7
      tests: value/negation mismatch, different-topic & restatement neutral, KG
      edge, injected NLI judge.)

### 3.2 — Claim-to-source span linking [C5]

- [x] Extend 1.2 to return fine-grained **span** attributions (which sentence
      span is supported by which source span), not document-level.
      (`claims/span-linker.ts`: `bestSupportingSpan` locates the source sentence
      span with max claim overlap, exact char offsets; `verifyOneClaim` now
      attaches `sourceSpan`/`spanText` to each `ClaimEvidence`.)
- [x] Tests: attribution points to the correct source span on a fixture. (8
      tests: exact-offset sentence split, best-span selection both directions,
      no-break fallback, claim-verifier integration; existing 12 claim tests
      stay green.)

### 3.3 — Uncertainty calibration & abstention [C6]

- [x] `verification/src/calibration/`: calibrate verifier/judge outputs
      (linear-probe or conformal) to emit confidence intervals; abstain/escalate
      below threshold [arXiv:2512.22245, arXiv:2407.18370].
      (`uncertainty-calibration.ts`: `expectedCalibrationError` (ECE) +
      `fitHistogramCalibrator` (bin→empirical-accuracy map) +
      `decideWithCalibration` which abstains when calibrated confidence <
      threshold; composes with the §1.8 conformal selective threshold.)
- [x] Tests: calibration reduces ECE on a fixture; low-confidence items abstain.
      (7 tests: ECE 0.15 on an over-confident fixture → 0.0 after calibration;
      cal(0.9)=0.6; an over-confident item abstains at a 0.7 bar.)

### 3.4 — Verifier-guided generation (best-of-N + verifier selector) [C7]

- [x] `verification/src/generation/verifier-guided.ts`: generate N candidates,
      select by the Phase-1 verifiers (factuality/faithfulness/STEM) with
      pessimistic selection; targeted self-refine on failed claims only.
      (`runVerifierGuidedGeneration`: scores each candidate through the
      `VerificationGate`, selects by `score − penalty·(1 − mean confidence)`
      preferring non-blocked, then refines a blocked selection from the gate
      critique.)
- [x] Wire as an opt-in generation mode in `content-generator`.
      (`ContentGenerator.generateBestOfNLesson`: varied-framing candidates +
      critique-targeted refine through the injected gate; fail-loud without
      one.)
- [x] Tests: verifier-guided selection picks the higher-factuality candidate on
      a fixture; refine fixes a planted unsupported claim. (4 runner tests + 2
      content-generator wiring tests.)

### 3.5 — Phase 3 gate

- [x] Eval deltas recorded; full suites green; tsc clean; lint 0; commit + push.
      (Phase 3 adds orthogonal depth capabilities — contradiction, span linking,
      ECE calibration, verifier-guided selection — none change the factuality
      gold-set baseline in `EVAL_BASELINES.md` (which scores the factuality
      verifier); no baseline delta applies. Full `@metis/verification` suite
      green; tsc clean; lint 0.)

---

# PHASE 4 — P1 Media (generative breadth + accessibility)

### 4.1 — Generative figures / concept images [M3]

- [x] Wire `integrations/src/yemaya-integration.ts` (and/or `@isis` image
      providers) from stub adapters to a real text→image generator for concept
      illustrations; gate output through the verification + critic stack.
      (`multimedia/src/image/concept-image-generator.ts`:
      `ConceptImageGenerator` over an injected `ImageProvider` seam — the caller
      wires `@isis`/Yemaya — with an injected `ImageValidator` gate (alt-text /
      VLM critic) that rejects a failing image;
      `ImageProviderNotConfiguredError` when absent.)
- [x] Tests (provider mocked at the boundary): a concept prompt yields an image
      artifact; missing provider fails loud. (4 tests incl. validator
      accept/reject.)

### 4.2 — Generative charts/graphs from data [M3]

- [x] `multimedia/src/diagram/data-figure-generator.ts`: auto-generate correct
      charts from a data spec (verified against the data); render via existing
      diagram pipeline. (`generateChartSpec` emits a Vega-Lite-style spec — tidy
      values, kind→mark, numeric/nominal x typing, color per series;
      `verifyChartSpec` proves the spec's values are exactly the input multiset
      (no fabricated/dropped point) and the mark matches the kind. The spec
      feeds a Vega-Lite/chart renderer.)
- [x] Tests: a data series produces the correct chart spec (assert
      encodings/values). (6 tests: encodings + values, single-series no-color,
      nominal x, tampered-value caught, mark mismatch caught.)

### 4.3 — Generative explainer video via provider seam [M4]

- [x] Add a text-to-video provider seam (Sora/Veo/Kling-class via
      `@isis`/Yemaya); use for segments where compositing/Manim is insufficient;
      fail-loud when unconfigured. (`video/text-to-video-provider.ts`:
      `TextToVideoGenerator` over an injected `TextToVideoProvider` seam (model
      hint sora/veo/kling), returns a `GeneratedVideoSegment` placed on the
      timeline; `VideoProviderNotConfiguredError` when absent.)
- [x] Tests (mocked): a segment request routes to the provider; result enters
      the pipeline. (3 tests: fail-loud, routed segment with start/end timing,
      model-hint passthrough.)

### 4.4 — Accessibility: auto alt-text [M5]

- [x] `multimedia/src/accessibility/alt-text-generator.ts`: VLM-generated
      alt-text for every diagram/figure/frame, then **verified** (does the
      alt-text match the image content?). (`AltTextGenerator` over an injected
      VLM `AltTextProvider` (fail-loud) + an `AltTextVerifier` — default
      `lexicalAltTextMatch` checks the alt-text mentions the image's known
      content labels.)
- [x] Make alt-text a required output; the media gate blocks artifacts
      missing/with-failing alt-text. (`altTextGate` → `passed:false` + the
      offending image ids when any alt-text is empty or non-matching.)
- [x] Tests: a figure gets alt-text; a wrong alt-text is rejected by the
      verifier. (8 tests: label-overlap match/mismatch/no-labels, fail-loud,
      generate+verify, gate block/pass.)

### 4.5 — Accessibility: audio descriptions + caption QA [M5]

- [x] Generate audio descriptions for visual-only content; add a caption-QA gate
      (timing, accuracy vs transcript) to `media_evaluation_service`.
      (`accessibility/caption-qa.ts`: `captionQa` checks
      ordering/overlap/bounds/ start<end + transcript content-token coverage;
      `AudioDescriptionGenerator` describes visual-only (un-narrated) segments
      via an injected provider, fail-loud when one is needed but absent.
      Consumed by media evaluation via import.)
- [x] Tests: mistimed/incorrect captions fail the QA gate. (9 tests: overlap,
      out-of-bounds, zero-length, accuracy-vs-transcript, audio-description
      needs/fail-loud/describe-visual-only.)

### 4.6 — Phase 4 gate

- [x] Full suites green; tsc clean; lint 0; commit + push. (Full
      `@metis/multimedia` suite green incl. the new image/chart/video/alt-text/
      caption-QA modules; tsc clean; lint 0. The 4.1-yemaya-adapter remediation,
      4.3/4.4 real provider+VLM wiring, and rendered-media checks remain seam
      swaps for a provider-configured host.)

---

# PHASE 5 — P2 (pedagogy & interactivity)

### 5.1 — Pedagogical / curriculum correctness [C8]

- [x] Age-appropriateness + reading-level fit to target band (reuse
      reading-level). (`pedagogy/pedagogy-verifier.ts` `fleschKincaidGrade` +
      target-band tolerance check.)
- [x] Prerequisite-consistency: a lesson doesn't depend on concepts marked
      un-taught (tie to `@metis/knowledge-graph` prerequisite chains + the
      adaptive engine). (`conceptPrerequisites` + `taughtConcepts` config —
      flags any present concept whose prerequisites are not in the taught set;
      the caller supplies the KG prerequisite chains.)
- [x] Misconception checks: wire `agents/src/agents/misconception-graph.ts` into
      the gate (flag content that reinforces known misconceptions). (Injected
      `misconceptions` patterns (literal or regex) — the caller wires the
      misconception-graph's patterns; matches are flagged.)
- [x] Tests: a lesson using an un-introduced prerequisite is flagged; a
      misconception-laden passage is flagged. (8 tests: FK grade spread,
      reading-band in/out, prerequisite missing/satisfied, misconception
      literal+regex, clean pass.)

### 5.2 — Interactive simulations / widgets [M6]

- [x] `multimedia/src/interactive/simulation-generator.ts`: parameterized
      PhET/Desmos-class interactive widgets per concept (spec → embeddable
      widget). (`generateFunctionWidget` auto-extracts free parameters from an
      expression into bounded sliders (Desmos-class); `generateSimulationWidget`
      builds from a PhET-class template library (projectile/pendulum/SHM),
      fail-loud on an unknown concept; `validateWidgetSpec` checks every slider
      is bounded and every plotter parameter is used.)
- [x] Tests: a concept yields a valid, parameterized widget spec. (8 tests:
      parameter extraction, function widget + ranges, simulation template,
      unknown-concept fail-loud, validation of unused-param / out-of-range.)

### 5.3 — Adaptive narration pacing [M7]

- [x] Replace templated narration timing with pacing learned from learner
      attention/comprehension signals (tie to adaptive telemetry).
      (`audio/adaptive-pacing.ts`: comprehension drives speaking rate (linear
      model, clamped), attention drives inter-segment pauses;
      `fitComprehensionResponse` learns the comprehension→wpm coefficients by
      OLS over telemetry, replacing the fixed templated wpm.)
- [x] Tests: pacing adjusts for a low-comprehension signal on a fixture. (7
      tests: low comprehension → slower wpm + longer segment, clamping, low
      attention → longer pause, OLS recovers generating intercept/slope.)

---

# PHASE 6 — Hardening, rollout & verification

### 6.1 — Cost / latency budgets (production)

- [x] Tune per-mode budgets (fast/balanced/deep/exhaustive) for verification +
      media; assert p95 latency + cost ceilings; kill-switch on overrun.
      (`hardening/budget-ceiling.ts`: `checkBudgetCeilings` computes
      nearest-rank p95 latency + total cost vs per-mode `MODE_CEILINGS` and
      recommends the kill-switch on breach, over the per-call
      `ComputeBudgetMeter` from 0.7. The ceiling _values_ are tuned from
      production telemetry; the assertion + kill-switch are here. 6 tests.)

### 6.2 — Drift & champion-challenger on verification quality

- [x] Monitor verifier agreement-with-human over time (Welch z drift); promote
      improved verifier/judge configs via a two-proportion z-test gate (reuse
      shared engine pattern). (`hardening/verification-drift.ts`:
      `monitorAgreementDrift` reuses the shared `detectMetricDrift` (Welch z) on
      the agreement signal; `evaluateVerifierPromotion` uses a two-proportion
      z-test to promote a challenger only when significantly better. 7 tests.)

### 6.3 — Production deployment (carries the graph-ML residue too)

- [ ] Point `ApiEmbeddingProvider` / NLI / PRM / Manim / image / video at live
      endpoints with credentials (fail-loud seams already exist).
      (DEPLOY-INFRA-BLOCKED on this box — no live endpoints/creds. Every seam is
      built and fail-loud: retriever/entailment (`@metis/knowledge-graph` +
      NLI), `ManimClient`, `ImageProvider`, `TextToVideoProvider`, `VlmCritic`,
      `AltTextProvider`, `AudioDescriptionProvider`, judge panel. **Manim is now
      a concrete integration, not just a seam**: the `services/metis` render
      worker (`metis.media.manim_render_*`) is built + tested against real
      Manim, with `createHttpManimTransport` /
      `createChildProcessManimTransport` connecting the `ManimClient` to it
      (§2.1); a render host points the transport at the deployed worker. The
      remaining providers are still credential/endpoint wiring.) _Re-read
      2026-09-18: "no live endpoints/creds on this box" is only partly true, and
      CLAUDE.md says keys are not missing. What an agent can do now: bind the
      embedding, NLI, VLM-critic, alt-text and audio-description seams to
      OpenRouter with the repository's cheap binding for the development and
      test stacks (`deepseek/deepseek-v4-flash-0731` for text; the cheapest
      vision model the catalogue lists for the critic), each through the seam's
      own configuration and failing loud when the key is absent; run the Manim
      worker locally (it is already built and tested against real Manim —
      install Manim in the `services/metis` environment if it is missing); and
      leave the image and text-to-video providers on `not_configured` until an
      Isis lane is chosen for them, recording that choice here. What is not an
      agent's: production credentials and endpoints. **Verify:** one integration
      spec per bound seam that runs live when `OPENROUTER_API_KEY` is set and
      skips loudly otherwise, with the measured cost; no seam returns a
      fabricated result without a key._
- [ ] Run the pgvector migration (`KNOWLEDGE_GRAPH_MIGRATION`) in the real
      `metis` database via the deploy pipeline. (DEPLOY-INFRA-BLOCKED: needs the
      prod DB + deploy pipeline.) _Board tag 2026-09-18: the migration is
      rehearsed locally already; running it in the production database is a
      deploy event, and Metis is a V1.2 room._ `blocked:release`
- [ ] Adopt the verification gate + agentic-media loop from the request-time
      host (`apps/metis/api-gateway` or a TS BFF / the Python service) so it
      runs in production, not just in libs. (DEPLOY-INFRA-BLOCKED: the gate
      (`composeP0Gate`/`runVerifiedGeneration`) and loop (`runAgenticMediaLoop`)
      are wired into `ContentGenerator`; adoption from the request-time host is
      a deploy step.) _Re-read 2026-09-18: this is code, not a deploy step. Make
      the request-time host call the gate: find where `apps/metis/api-gateway`
      (or the Python service behind it) constructs `ContentGenerator`, route its
      generation endpoint through `runVerifiedGeneration` with `composeP0Gate`,
      and surface the gate's verdict and the loop's attempts in the response.
      **Verify:** a route-level spec with the providers doubled at their
      boundary shows a failing gate blocks the response and a passing one
      returns it with its evidence; a grep shows no generation route that
      bypasses the gate._

### 6.4 — Docs & runbooks

- [x] Update `DOMAINS/metis/*` (architecture/features/specifications) with the
      verification + agentic-media subsystems. (`architecture.md` gains a
      "Correctness Verification & Agentic Media" section (gate + verifiers,
      Planner→Coder→Critic loop, sandboxed Manim render worker); `features.md`
      gains §14 Content Correctness Verification + §15 Agentic Teaching Media;
      `specifications.md` gains §12 Correctness Verification Subsystem (gate
      decision algebra, verifier table, eval baselines) + §13 Agentic Media &
      Manim Render Service (loop, render request/response + sandbox contract,
      transports, segment composition). Regenerated the three
      `DOMAINS/metis/*.html` via `tools/render-domain-docs.py`. The in-lib
      `DECISION.md`/`EVAL_BASELINES.md`/`MEDIA_EVAL_BASELINES.md`/`RUNBOOKS.md`
      remain the deep references. Note: the regen also date-bumps every other
      domain's HTML footer — those unrelated changes were reverted to keep this
      scoped to metis.)
- [x] Runbooks: how to add a verifier, a media agent, a gold set; how HITL
      feedback flows. (`libs/metis/verification/RUNBOOKS.md`.)

### 6.5 — Final adversarial verification pass

- [x] Repo-wide adversarial stub scan over all new code (the mandated grep set);
      every hit read in context; zero result-faking. (Scanned all 30 impl files
      changed this session; clean. Only 2 hits, both PRE-EXISTING in
      `content-generator.ts` from commit `bc13f0788e` — a `parseLessonSections`
      doc comment and a `'metis-simulated'` provenance label, neither
      result-faking.)
- [x] Confirm the fact-checking-agent no longer fabricates verdicts (0.4)
      anywhere callers use it. (Confirmed: reports `unverified` (honest absence)
      with no verifier; only produces truth verdicts via an injected
      evidence-grounded verifier — never fabricates.)
- [x] Full metis suites green; lint 0 errors; final commit + push (branch +
      main). (`@metis/verification` 161 tests, full multimedia 853, agents 885
      tests — all green; the sole agents file-fail is the pre-existing,
      unrelated `curriculum-agent.spec.ts` `@oshun/contracts/metis` alias gap
      from another session. Lint 0 errors on all new code.)

---

## Cross-references (SOTA → task)

| SOTA method                                                 | Source                 | Tasks                                        |
| ----------------------------------------------------------- | ---------------------- | -------------------------------------------- |
| FActScore atomic-fact support                               | arXiv:2305.14251       | 1.1, 1.2                                     |
| SAFE / F1@K                                                 | arXiv:2403.18802       | 1.1, 1.2                                     |
| FaStfact chunk-level extraction                             | arXiv:2510.12839       | 1.1                                          |
| TRACe / RAGBench faithfulness (DeBERTa)                     | arXiv:2407.11005       | 1.3, 1.4                                     |
| Citation placement (retrieval-driven; post-hoc vs gen-time) | arXiv:2509.21557       | 1.4                                          |
| ThinkPRM process reward model                               | arXiv:2504.16828       | 1.7                                          |
| Judge position bias                                         | arXiv:2406.07791       | 1.8                                          |
| Agreeableness bias + ensembles (minority-veto, regression)  | arXiv:2510.11822       | 1.8                                          |
| Cascaded Selective Evaluation (conformal)                   | arXiv:2407.18370       | 1.8, 3.3                                     |
| IRT/GRM judge reliability                                   | arXiv:2602.00521       | 1.8                                          |
| Linear-probe judge calibration                              | arXiv:2512.22245       | 1.8, 3.3                                     |
| TheoremExplainAgent (Manim video)                           | arXiv:2502.19400       | 2.1–2.6                                      |
| Code2Video (Planner→Coder→Critic, TeachQuiz)                | arXiv:2510.01174       | 2.2–2.6                                      |
| Sora 2 / Veo 3.1 / Kling 3 (text-to-video)                  | 2026 model comparisons | 4.3                                          |
| Hedra Character-3 / HeyGen Avatar 5 / OmniHuman v1.5        | 2026 model comparisons | (existing avatar pipeline; provider upgrade) |

> Verify each task by reading the code you wrote and running its tests — not by
> file existence or this table. Mark `[x]` only after the per-task gate passes
> in-session.