Metis — Correctness Verification & Agentic Teaching-Media: SOTA Gap Analysis

Date: 2026-06-16 Scope: the metis domain (libs/metis/**, services/metis/**, apps/metis/**) — an adaptive-learning / educational-content platform. Question (as set): analyze metis as it stands today and identify missing features, especially around (1) SOTA knowledge & correctness verification and (2) creation of rich, beautiful, compelling media with autonomous agentic systems to teach concepts. Method: (a) source-level audit of metis (real-vs-stub, file:line); (b) a fan-out / adversarially-verified deep-research pass on the 2025–2026 SOTA for correctness verification (23 verified findings) + a targeted pass on agentic teaching-media; (c) cross-reference with the repo's existing platform reports (V1_V9_AGENTIC_CONTENT_SOTA_ASSESSMENT_2026-06-14.md, AGENTIC_CONTENT_QUALITY_ANALYSIS_2026-06-13.md).

0. Bottom line#

Metis is strong where most ed-tech is weak — real, end-to-end multimodal media orchestration (TTS with loudness normalization, video compositing, avatar + lip-sync, diagrams, presentations, multilingual narration) and real grounded governance (lesson-revalidation on source retraction, a rights/freshness filter, GraphRAG source inspection, a citation-required lesson schema). But it is weak exactly where education should be strongest: correctness verification is heuristic and single-pass, its named "fact-checker" does not actually check facts, and it generates no truly generative media (no math animation, no figure/image generation, no agentic author→critic→media loop).

Two findings dominate everything:

1. Education is the verifiable domain — and metis under-exploits that. The platform's own quality analysis nails the asymmetry: verifier-guided methods (best-of-N, process reward models, claim verification) deliver their big gains where a ground-truth verifier exists — math, code, formal reasoning; creative content has no oracle (AGENTIC_CONTENT_QUALITY_ANALYSIS_2026-06-13.md §4). Educational content is precisely that domain. Yet metis verifies content with document-level heuristics and a single uncalibrated LLM judge, while the 2026 SOTA for verifiable content — atomic-claim decomposition, RAG faithfulness scoring, and process reward models — is absent.

2. Metis does not ride the platform's verified SOTA content engine. A grep across libs/metis, services/metis, apps/metis for content-quality-judge / content-release-gates / content-service / grounding-gate / judge-panel returns nothing. The shared engine — a generate→judge-gate→refine loop with a calibrated, de-biased LLM-judge panel, best-of-N, a reward model with hacking detection, drift / champion-challenger, and a fail-loud canon-grounding gate that blocks ungrounded claims (V1_V9_AGENTIC_CONTENT_SOTA_ASSESSMENT_2026-06-14.md §2, 312+ tests) — sits in the same monorepo and metis reimplements a weaker subset of it.

Verdict by sub-area:

1. Current state of metis (verified)#

1.1 Correctness & verification — what's real, what isn't#

Real and good:

• Lesson revalidation on source retraction (research/src/credibility/lesson-revalidation.ts) — deterministic: counts credible supporting sources, applies a min-credibility threshold (≥0.6), disposes a lesson retained | flagged-for-review | quarantined. Wired to a veritas cascade contract. This is real, operational, and uncommon.
• Rights/freshness governance (knowledge-graph/.../governance.ts) — evaluateNode/Edge blocks blocked/unknown-rights artifacts in learner contexts, requires source-backing, honors license/allowed-use. The GraphRAG benchmark suite (applications/graph-rag-benchmark.ts) inspects retrieved nodes for credibility, staleness, invalidation, rights, and source diversity.
• Cited-lesson schema (llm-client/.../content-generator) — the output schema requires ≥1 citation per lesson, with citation extraction from the body.
• LLM-as-judge quality evaluator (llm-client/.../quality-evaluator.ts) — a real single LLM call grading 5 pedagogical dimensions (completeness, readability, structure, relevance, Bloom alignment) with Zod validation + heuristic fallback.
• Reading-level analysis (quality/.../reading-level.ts) — real Flesch-Kincaid / Gunning Fog / Coleman-Liau / SMOG with complexity detection and linting rules.
• Credibility scoring (research/.../credibility.ts) — heuristic 5-factor with an exponential currency-decay by domain half-life.

The problem areas (verified by reading the code):

• The "fact-checking agent" does not check facts. agents/.../fact-checking-agent.ts makes no LLM call and no retrieval/search. It heuristically classifies a sentence's claim type, derives a score, maps it to a verdict (verified / disputed / false / …) via determineVerdict(...), and returns hardcoded generic source categories (statistical: ['Government statistics databases', 'Published research studies']). It emits authoritative-sounding verdicts it never grounded in evidence — a correctness / credibility hazard, and exactly the result-faking pattern the repo's quality rules forbid.
• Math validation is syntax-only. quality/.../math-validator.ts checks balanced brackets / well-formed LaTeX (tagged ['math','syntax']). There is no symbolic / CAS / step / correctness checking — in the one place education has a true oracle.
• The quality judge is single-pass and uncalibrated. One isis.complete() call; no panel, no position-bias mitigation, no calibration against human gold — the very failure modes the 2026 judge literature flags (see §2.1).
• Citations are required but never verified to support their claims. quality/.../citation-checker.ts validates citation format (APA/MLA), not whether a cited source actually entails the sentence. Citation existence ≠ citation sufficiency.
• Search/embeddings/corpus/citations modules in research/ are largely types/factories (interfaces + constructors), not implementations.

1.2 Teaching media — what's real, what's absent#

Real and strong (this is metis's best work):

• Audio/TTS (multimedia/src/audio/*): real TTS orchestration (SSML, provider abstraction incl. ElevenLabs/Chatterbox/local, emotion-driven prosody, ffmpeg format conversion), ITU-R BS.1770 LUFS loudness normalization, silence/fingerprint/dup detection, narration planning, phonetic customization, multilingual narration.
• Video (multimedia/src/video/*): frame-level compositing, transitions, captions, GPU worker pool, avatar generation + phoneme→viseme lip-sync, chroma-key/PiP/audio-mix compositor.
• Diagrams (multimedia/src/diagram/* + agents/diagram-agent.ts): SVG rendering with theming, progressive-reveal animation, interactive tooltips, Mermaid/SVG/JSON output.
• Presentations & lectures (multimedia/src/presentation/*, lecture-generation/*, avatar-teacher/*): deck assembly, speaker notes, multi-format export (PDF/HTML/MP4), end-to-end lecture orchestration (slides + narration + diagrams + transcripts), avatar teacher synthesis. The Python service adds lecture_generation_service and a media_evaluation_service (audio/video/caption/sync/embodiment quality).

Absent vs SOTA:

• No generative figure/diagram-from-meaning — diagrams are rendered from caller-specified specs; there is no text→image generation for concept illustrations (would route via @isis/Yemaya — currently only stub adapters in integrations/yemaya-integration.ts).
• No programmatic math/CS animation (Manim-class). The single highest-value ed-media gap.
• No generative (pixel) text-to-video (Sora/Veo-class) — only compositing of supplied assets.
• No interactive simulations/widgets (PhET/Desmos-class).
• No multi-agent author→critic→media refinement loop. Agents are one-pass generators; there is no plan→generate→render→critique→repair cycle.
• No accessibility automation — transcripts/captions exist, but no auto alt-text for diagrams/figures, no audio descriptions, no sign-language track.

2. The 2026 SOTA yardstick (cited)#

2.1 Correctness & knowledge verification#

• Atomic-claim decomposition is the factuality paradigm. Break a generation into atomic facts, verify each against a knowledge source, aggregate. FActScore = % of atomic facts supported (automated retrieval+LLM estimator <2% error vs humans on biographies; higher cross-domain) [arXiv:2305.14251]. SAFE verifies each atomic fact via multi-step Google-Search reasoning and scores F1@K (precision of supported facts vs recall to a target length K) [arXiv:2403.18802]. FaStfact (EMNLP 2025) cuts extraction from O(N) to O(N/w) via chunk-level extraction + confidence pre-verification, with the tightest human alignment among evaluated pipelines [arXiv:2510.12839].
• LLM-as-judge is unreliable unless engineered. Systematic position bias varying by judge/task [arXiv:2406.07791] and an "agreeableness" bias — judges confirm correct outputs at TPR >96% but reject invalid ones at TNR <25% [arXiv:2510.11822]. SOTA mitigations: Cascaded Selective Evaluation (conformal, distribution-free guarantee of human-agreement by abstaining/escalating; cheap models exceed 80% agreement where GPT-4 gets 77.8%) [arXiv:2407.18370]; minority-veto and regression bias-correction ensembles (max error 2.8%→1.2%, beating a 14-LLM majority) [arXiv:2510.11822]; IRT/Graded-Response-Model reliability diagnostics — no judge is uniformly reliable across criteria, validate per task [arXiv:2602.00521]; linear probes on judge hidden states for calibrated confidence at ~10× less compute than self-consistency [arXiv:2512.22245].
• RAG faithfulness/groundedness has dedicated scorers. A fine-tuned DeBERTa-v3 NLI judge reaches hallucination-AUROC 0.64–0.86, beating prompted GPT-3.5 (0.51–0.65), RAGAS, TruLens; the TRACe framework (RAGBench) defines context-relevance, utilization, completeness, and adherence (= faithfulness/groundedness/attribution: are all parts of the response grounded in context?) [arXiv:2407.11005].
• Citations: retrieval is the dominant driver; placement is a trade-off. Zero-shot→RAG is the biggest citation-correctness gain (FEVER 27%→77%); post-hoc citation maximizes coverage, generation-time maximizes precision [NeurIPS 2025, arXiv:2509.21557].
• STEM correctness uses process reward models. ThinkPRM — a generative long-CoT verifier fine-tuned on ~1% of PRM800K labels — beats discriminative PRMs trained on the full set [arXiv:2504.16828]. PRM800K is the canonical process-supervision dataset.

2.2 Autonomous agentic teaching-media generation#

• Programmatic, agent-authored math/CS animation is the education-native SOTA. TheoremExplainAgent generates long-form (>5 min) theorem-explanation Manim videos; on TheoremExplainBench (240 theorems, 5 metrics) an o3-mini agent hits 93.8% success / 0.77 — and crucially finds that multimodal video explanations expose reasoning gaps that text-only explanations hide [arXiv:2502.19400].
• Multi-agent author→critic→media is the pipeline pattern. Code2Video uses a Planner → Coder → Critic trio to generate educational videos via executable code (not pixel synthesis), with a VLM critic refining layout/clarity; +40% over direct code generation, comparable to human tutorials, evaluated on MMMC with the TeachQuiz metric (does a VLM learn from the video?) [arXiv:2510.01174].
• Generative video is production-ready in 2026. Sora 2 (physics realism, native audio), Veo 3.1 (4K, native audio), Kling 3, Seedance 2.0 — single-API, multi-shot, native audio.
• Talking-head avatars are omnimodal. Hedra Character-3 (joint image+text+audio), HeyGen Avatar 5 (15-s training, 1080p, 175+ languages), ByteDance OmniHuman v1.5 (film-grade); joint audio-video models beat separate-step lip-sync (EvalTalker, arXiv:2512.01340). (Metis already has avatar+lip-sync orchestration — the gap is the generative model behind it, not the pipeline.)
• Baseline vs SOTA distinction: baseline = template/compositing of supplied assets; SOTA = an agent that plans a lesson, generates the figures/animation/voiceover, renders, and a critic loops until a learning-outcome proxy (e.g. TeachQuiz) passes — with accessibility (alt-text, captions) produced as a first-class output.

3. Gap analysis — Area 1: correctness verification (prioritized)#

4. Gap analysis — Area 2: rich agentic teaching media (prioritized)#

5. The cross-cutting gap (ties both areas together)#

Both areas share one root cause: metis runs one-pass agents and its own quality subset instead of the platform's verified, closed-loop SOTA engine. The shared generate → judge-gate → (regenerate | refine) → re-gate loop, the calibrated judge panel, the reward model with hacking detection, drift/champion-challenger, and the canon-grounding gate that blocks ungrounded claims are real and tested (V1_V9_AGENTIC_CONTENT_SOTA_ASSESSMENT_2026-06-14.md §2) — and unused by metis. The same loop is exactly what both a correctness-verifier (C1–C4, C7) and an author→critic→media pipeline (M1–M2) need. The single highest-leverage move is to put metis content and media generation on that loop, with education-specific verifiers (claim/RAG/PRM/CAS) plugged in as the gate — turning "generate then a human checks" into "generate, machine-verify against ground truth, refine, then a human handles only low-confidence cases."

The platform quality report's standing warnings apply directly to metis: don't scale generation before the quality/verification layer exists; and the human reviewers' accept/ reject decisions should train the verifier (active learning), not be warehoused.

6. Prioritized roadmap#

• P0 (correctness-critical, do first): C1 (claim-decomposition verifier replacing the fake fact-checker) · C2 (RAG faithfulness/citation-sufficiency) · C3 (real math/code/step correctness) · C4 (calibrated judge panel — reuse the shared engine) · M1 (agentic Manim animation) · M2 (author→critic→media loop). Sequence: stand the verifiers up as a gate, put generation on the shared loop, then add the Manim media agent behind the same gate.
• P1: C5–C7 (contradiction/span linking, calibration, verifier-guided best-of-N) · M3–M5 (generative figures, explainer video via Isis/Yemaya, accessibility automation).
• P2: C8 (pedagogical/curriculum correctness via the KG + misconception-graph) · M6–M7 (interactive sims, adaptive pacing).

7. What metis genuinely gets right (so this isn't only critique)#

Real, end-to-end multimodal media orchestration (TTS + LUFS audio mastering + video compositing + avatar/lip-sync + diagrams + decks + multilingual lectures) is uncommon and production-shaped. Grounded governance — lesson revalidation on source retraction, a context-aware rights/freshness filter, GraphRAG source inspection, and a citation-required lesson schema — is ahead of typical ed-tech and is the right foundation to bolt real verification onto. The gap is not the plumbing; it's the verification intelligence and the generative-media intelligence that should ride on top of it.

Appendix — sources & caveats#

Area 1 (3-vote verified): FActScore arXiv:2305.14251 · SAFE/F1@K arXiv:2403.18802 · FaStfact arXiv:2510.12839 · position bias arXiv:2406.07791 · agreeableness bias + ensembles arXiv:2510.11822 · Cascaded Selective Evaluation arXiv:2407.18370 · IRT judge diagnostics arXiv:2602.00521 · linear-probe calibration arXiv:2512.22245 · RAGBench/TRACe arXiv:2407.11005 · citation placement arXiv:2509.21557 · ThinkPRM arXiv:2504.16828. Area 2 (targeted pass; the automated deep-research run failed to verify Area-2 claims — these are from primary papers + 2026 model comparisons, treat model-ranking deltas as directional): TheoremExplainAgent arXiv:2502.19400 · Code2Video arXiv:2510.01174 · talking- head eval arXiv:2512.01340; Sora 2 / Veo 3.1 / Kling 3 / Hedra Character-3 / HeyGen Avatar 5 / OmniHuman v1.5 per 2026 vendor/benchmark comparisons. Internal evidence: libs/metis/agents/src/agents/fact-checking-agent.ts (heuristic verdicts, no retrieval/LLM, hardcoded source categories), quality/.../math-validator.ts (syntax-only), llm-client/.../quality-evaluator.ts (single isis.complete), research/.../credibility/lesson-revalidation.ts, knowledge-graph/.../governance.ts, multimedia/src/{audio,video,diagram,presentation}/*, integrations/src/yemaya-integration.ts (stub adapters); grep confirms metis does not import content-quality-judge / content-release-gates / content-service. Caveats: several SOTA headline numbers are authors' self-reports on their own benchmarks (FaStfact, Code2Video, Cascaded Selective Evaluation); agreeableness-bias / ensemble magnitudes derive from one narrow task; how these verifiers perform on pedagogical correctness (vs biography/code domains) is an open question; the Area-2 automated verification did not complete, so its system rankings are directional.