Date: March 23, 2026
Author: Codex
Scope: docs/releases/v1/specs/todos.md,
docs/releases/v1/reports/unified-product-gap-analysis-and-nisaba-integration-report-2026-03-22.md,
docs/releases/v1/domains/metis/*,
docs/domains/metis/extras/codebase-analysis/*, apps/metis/*, libs/metis/*,
services/metis/*, minerva/*, and current official external product/docs
pages accessed on March 23, 2026
1. Executive Summary#
The canonical OSHUN V1 analysis and backlog are incomplete because they omitted
the metis domain from beginning to end.
That omission is material, not cosmetic.
Metis is not a small side module. The repo already contains:
- a standalone Metis web app
- a standalone Metis admin app
- a Metis API gateway
- a Metis Python service
- a Metis worker
- a large
libs/metis/*ecosystem - Minerva-origin analysis documents and migration plans
- a separate Oshun-local Metis phase document with 613 checklist items
But none of that is represented in the canonical March 22, 2026 OSHUN V1
backlog. docs/releases/v1/specs/todos.md contains zero metis references.
The second major finding is that Metis currently sits in an unstable middle state:
- the repo contains real domain libraries, route handlers, schemas, and tests
- the main user-facing Metis web/admin surfaces are still largely mock-driven
- the tutoring backend is a thin prompt wrapper, not yet a grounded adaptive tutor
- the canonical Oshun shell/admin/domain contracts do not yet treat Metis as a first-class V1 concern
The third major finding is that "full Metis integration" cannot mean just "expose the Metis app." To be state-of-the-art in 2026, Metis needs to operate as:
- OSHUN's materials-first educational authoring and delivery layer
- a grounded source-aware course generation system
- a real adaptive tutor with memory, mastery state, and assessment feedback
- a multimodal delivery system spanning text, diagrams, audio, video, and natural avatar instructors
- a governed educational platform with rights, provenance, quality, policy, and LMS interoperability
Bottom line:
- The current OSHUN V1 canonical backlog is incomplete until Metis is added.
- The current Metis implementation is promising but not yet "complete and total integration."
- The highest-leverage path is to make Metis a first-class Oshun domain and a shared educational substrate connected deeply to Sophia, Isis, Lilith, Iris, Psyche, Yemaya, Themis, and the Oshun shell/admin stack.
2. Canonical Planning Correction#
2.1 What the March 22 canonical analysis missed#
The March 22, 2026 canonical report explicitly scoped:
apps/oshun/*libs/oshun/*libs/nisaba/*apps/sophia/*libs/sophia/*- several other Oshun domains and integrations
It did not scope:
apps/metis/*libs/metis/*services/metis/*docs/domains/metis/extras/*docs/releases/v1/domains/metis/*minerva/*
That omission propagated into the canonical backlog:
docs/releases/v1/specs/todos.mdhas zerometisreferences- the backlog derives from the March 22 report, so the omission is structural
- Metis therefore has no canonical V1 program status, no canonical release gates, and no canonical cross-domain completion criteria in the main backlog
2.2 Why this matters#
Metis is one of the few repo areas that already spans:
- authoring
- content generation
- tutoring
- adaptive learning
- assessment
- multimedia
- LMS interoperability
- domain-to-domain knowledge transfer
If Metis is omitted from V1 planning, then OSHUN V1 is missing its educational platform layer and several important cross-domain leverage points.
3. Research Method#
3.1 Repository evidence reviewed#
Primary repo evidence for this report:
docs/releases/v1/specs/todos.mddocs/releases/v1/reports/unified-product-gap-analysis-and-nisaba-integration-report-2026-03-22.mddocs/releases/v1/domains/metis/README.mddocs/releases/v1/domains/metis/feature-inventory.mddocs/releases/v1/domains/metis/implementation-map.mddocs/domains/metis/extras/codebase-analysis/01-python-modules.mddocs/domains/metis/extras/codebase-analysis/02-agents-taxonomy.mddocs/domains/metis/extras/codebase-analysis/05-multimedia-pipeline.mddocs/domains/metis/extras/codebase-analysis/06-linters-quality-control.mddocs/domains/metis/extras/codebase-analysis/07-research-learning-database-cli.mdminerva/AUDIT_1.mdminerva/AUGGIE_REPORT_CONTENT_GENERATION.mdapps/metis/web/*apps/metis/admin/*apps/metis/api-gateway/*apps/metis/worker/*libs/metis/*services/metis/*
3.2 External benchmark set#
The external benchmark uses current official product/docs pages accessed on March 23, 2026. The benchmark is intentionally product-operational rather than research-only. It focuses on what working systems now expose to customers and developers.
Primary benchmark categories:
- AI study/tutor experiences: OpenAI Study Mode, Khanmigo, Duolingo Max, Gemini study materials
- BYOM/source-grounded learning workflows: NotebookLM, Gemini file-backed study artifacts
- natural voice/avatar delivery: OpenAI Realtime/WebRTC, Tavus, HeyGen, ElevenLabs, Synthesia
- LMS/security standards: 1EdTech LTI 1.3 / LTI Advantage
3.3 Interpretation rule#
This report distinguishes three states:
- verified implementation: directly supported by current repo evidence
- planning claim: stated in repo docs/checklists but not fully supported by current implementation evidence
- inference: a recommendation derived from repo evidence plus external benchmarks
4. Verified Current State of Metis in This Repo#
4.1 What is clearly present#
Verified:
- Metis has a real repo footprint under
apps/metis/*,libs/metis/*, andservices/metis/*. - Metis has substantial type/domain libraries for course, assessment, tutoring, research, learning, multimedia, quality, model registry, prompt management, A/B testing, and cost tracking.
- Metis has a real API gateway and worker with tests.
- Metis has a Python backend with SQLAlchemy models, Alembic migrations, and FastAPI-style API modules.
- Metis has meaningful test coverage in the library/gateway/worker layers: 211
test/spec files were discoverable under
apps/metis,libs/metis, andservices/metis.
4.2 What is clearly not complete#
Verified:
apps/metis/web/*is still heavily mock-backed.apps/metis/admin/*is still heavily mock-backed.apps/metis/web/src/app/tutoring/page.tsxsimulates tutor replies locally.apps/metis/web/src/hooks/useAuth.tsstill uses a mock token and mock user.apps/metis/admin/src/lib/admin-client.tsreturnsMOCK_*data rather than a live backend.services/metis/src/metis/services/tutoring_service.pycurrently builds a generic system prompt and forwards the conversation to OpenAI/Anthropic; it does not implement grounded retrieval, mastery tracking, durable tutor memory, or a true pedagogical policy engine.- no Metis OpenAPI or proto contract root was discoverable under
openapi/orlibs/proto/ - no Playwright/E2E coverage was discoverable for
apps/metis/weborapps/metis/admin - no Python test files were discoverable under
services/metis
4.3 Planning claims versus verified implementation#
docs/releases/v1/domains/metis/feature-inventory.md is valuable as a scope
map, but it should not be treated as a completion ledger.
Why:
- it marks 613 checklist items as complete
- it claims broad cross-domain integration completion
- it claims success metrics already satisfied
- the current repo still shows mock-driven UX, thin tutoring behavior, absent shell-level integration, missing canonical backlog coverage, and missing web E2E evidence
Inference:
The phase-25 Metis document is best understood as an ambitious migration and integration scope inventory, not a verified production-readiness record.
4.4 Current-state summary matrix#
| Area | Current state | Assessment |
|---|---|---|
| Core TS libraries | Real and broad | Strong foundation |
| API gateway | Real and tested | Strong foundation |
| Worker/jobs | Real and tested | Strong foundation |
| Python service | Real scaffold, limited evidence of production maturity | Partial |
| Web learner app | Mostly mock data and simulated flows | Incomplete |
| Admin app | Mostly mock data and local fake operations | Incomplete |
| Tutoring intelligence | Generic prompt wrapper, not true adaptive grounded tutor | Incomplete |
| BYOM ingestion | Strong legacy design in Minerva analysis, weak verified Oshun runtime evidence | Partial |
| Multimedia/avatar delivery | Good architectural intent, limited verified integrated runtime evidence | Partial |
| Oshun shell integration | Missing from canonical shell/backlog | Missing |
| LMS/export standards | Present in scope docs, not yet canonically verified in V1 | Partial |
| QA and release gates | Strong unit-test posture in libs, weak user-flow and service verification | Partial |
5. State-of-the-Art Benchmark: What "Good" Looks Like in 2026#
5.1 Content generation and learning artifacts#
Current benchmark behaviors from official sources:
- OpenAI Study Mode now uses Socratic questioning, adaptive difficulty, open-ended understanding checks, memory-aware personalization, and uploaded materials such as PDFs/images.
- Gemini supports creating quizzes, flashcards, and study guides from prompts and uploaded materials.
- NotebookLM is explicitly source-first: each notebook is built from a source set, and the chat works against those sources.
- Synthesia positions AI video as multilingual, SCORM-aware training content infrastructure rather than just "avatar clips."
Inference:
State-of-the-art educational content generation is no longer "generate a course." It is:
- materials-first
- source-grounded
- artifact-rich
- revision-friendly
- exportable into LMS ecosystems
- measurable after delivery
5.2 AI tutor behavior#
Current benchmark behaviors from official sources:
- OpenAI Study Mode asks interactive questions to establish goals and skill level, then teaches toward the answer rather than only giving the answer.
- Khanmigo is explicitly positioned as a virtual tutor using hints, questions, and explanations, with teacher/parent visibility and moderation workflows.
- Duolingo Max's Video Call keeps real-time conversation going and remembers the discussion for later sessions.
Inference:
State-of-the-art AI tutoring in 2026 requires:
- guided reasoning instead of answer dumping
- persistent learner context
- mastery-aware adaptation
- teacher/guardian oversight where relevant
- built-in safety and anti-cheating policies
- multimodal turn-taking, not only text chat
5.3 BYOM#
In this codebase, BYOM means Bring Your Own Materials.
Current benchmark behaviors from official sources:
- NotebookLM organizes learning around explicit source collections
- Gemini study-material workflows can use uploaded files as context
- OpenAI Study Mode can work from uploaded learner materials
Inference:
State-of-the-art BYOM is not just file upload. It includes:
- ingestion across many formats
- source scoping at notebook/course/workspace level
- citation/provenance retention
- permissions and tenancy controls
- source-linked tutoring and assessment generation
- refresh/version handling when source materials change
5.4 Natural avatar teachers, lecturers, and mentors#
Current benchmark behaviors from official sources:
- OpenAI recommends WebRTC for browser speech-to-speech apps and provides VAD and semantic turn detection for natural voice interaction.
- Tavus recommends its full pipeline for perception, turn-taking, and rendering as one conversational system.
- HeyGen positions LiveAvatar as the production version of its Interactive Avatar stack.
- ElevenLabs positions its TTS API around low latency and high-fidelity speech.
- Synthesia positions avatar video as multilingual, on-brand, SCORM-capable training delivery.
Inference:
State-of-the-art avatar educators require two distinct modes:
-
Pre-rendered instructional delivery
- polished lecturer videos
- multilingual dubbing
- LMS packaging
- brand/style controls
-
Interactive real-time instruction
- low-latency turn-taking
- interruption handling
- memory and context carry-over
- visually coherent avatar rendering
- voice naturalness and persona consistency
5.5 Standards and enterprise interoperability#
Current benchmark behaviors from official sources:
- 1EdTech explicitly recommends LTI 1.3 / LTI Advantage as the secure modern teaching-and-learning integration baseline, replacing older LTI variants.
- 1EdTech positions OneRoster as the standard way to exchange roster information, course materials, and grades between school systems.
- 1EdTech positions QTI 3 as the portable, accessible assessment-content format with support for CAT and richer interoperability.
- 1EdTech positions CASE, Open Badges, and CLR as core standards for competency-aligned and portable learner records.
- 1EdTech positions Caliper as the standard analytics vocabulary for institution-facing learning data interoperability across tools.
Inference:
Metis cannot be considered fully integrated or enterprise-grade without:
- LTI 1.3 / LTI Advantage
- xAPI/cmi5-grade learning telemetry strategy
- OneRoster-grade rostering and gradebook interoperability where institutional programs are promised
- QTI-grade assessment portability and CAT interoperability where adaptive or portable assessment flows are promised
- competency and learner-record interoperability through CASE / CLR / Open Badges where standards-based achievement portability is promised
- SCORM fallback where required
- clear content identity, versioning, and grade passback contracts
6. Gap Analysis#
6.1 Content generation gap#
Current repo strengths:
- rich Minerva-origin multi-agent architecture
- strong linter architecture
- course hierarchy and assessment models
- meaningful multimedia and diagram concepts
Current gap:
- learner-facing authoring flow is still mock-heavy
- no canonical source-grounded course build pipeline is verified end-to-end in Oshun
- no canonical operator workflow for approving source packs, course builds, generated assets, and publication bundles
- no verified canonical export contract for LMS-facing deliverables
Recommendation:
Metis content generation should be redefined as a governed build system:
materials -> grounded outline -> lesson assets -> tutoring hooks -> quality gates -> rights/provenance bundle -> LMS/web/mobile delivery
6.2 AI tutor gap#
Current repo strengths:
- tutoring domain types and service scaffolding exist
- Minerva analysis documents show deep ambition around affect, style, mastery, and scaffolding
- Metis has related libs for learning, assessment, and tutoring
Current gap:
- current tutoring implementation is essentially provider call + generic prompt
- no durable learner memory loop
- no source-grounded tutor answer path
- no strong assessment-to-tutor feedback loop
- no tutor persona governance or escalation model
Recommendation:
Metis tutoring should be rebuilt around a canonical tutoring policy engine that combines:
- Sophia grounding
- Lilith dialogue/pedagogy
- Iris learner memory
- Metis mastery state
- Psyche real-time voice/avatar delivery
6.3 BYOM gap#
Current repo strengths:
- Minerva analyses document multi-format ingestion for 30+ formats
- corpus, parsing, and research foundations exist in the legacy architecture
Current gap:
- no canonical Oshun-level BYOM contract is visible
- no explicit linkage between uploaded source materials and downstream tutor, quiz, asset, and certification outputs
- no verified rights/provenance pipeline for BYOM-driven generation
Recommendation:
Metis needs a first-class LearningSourceBundle and GroundedCourseBuild
architecture, not just "uploaded files."
6.4 Avatar educator gap#
Current repo strengths:
- strong architectural intent in Minerva/Metis docs
- existing Psyche and related avatar/realtime assets elsewhere in the repo
- multimedia abstractions already exist in
libs/metis/multimedia
Current gap:
- no verified end-to-end avatar educator stack in Metis itself
- no canonical distinction between prerecorded lecturers and live tutors
- no explicit persona, likeness-consent, rights, safety, and fallback policies
Recommendation:
Metis should support three educator modes:
- text-first tutor
- voice tutor
- avatar tutor / lecturer
Each mode should share the same pedagogical brain and memory system while using different renderers and latency budgets.
6.5 OSHUN-wide integration gap#
Current gap:
- Metis is not yet a canonical Oshun shell domain
- Metis is not yet present in the March 22 V1 scope model
- Metis operator surfaces are not integrated into a canonical Oshun admin model
- Metis cross-domain leverage is described in domain docs but not carried into the canonical V1 execution ledger
Recommendation:
Metis must be integrated at four layers:
- domain registry and shell
- shared services and contracts
- operator/review/admin systems
- cross-domain product flows
7. Required Full Integration Across OSHUN V1#
7.1 Shell and navigation#
Metis should become a first-class Oshun domain in:
- domain registry
- shared route typing
- auth/domain preference typing
- analytics enums
- notification preferences
- deep links
- assistant context
- global search and recommendation systems
- entitlement and subscription surfaces
7.2 Shared backend/service ownership#
Recommended ownership model:
- Metis: educational product logic, course/tutor/assessment orchestration
- Sophia: source ingestion, grounding, citations, evidence, knowledge graph
- Isis: model routing, prompt registry, experiments, cost, batch inference
- Lilith: pedagogical dialogue behavior, tutor persona logic, coaching tone
- Iris: learner memory, session continuity, preference persistence
- Psyche: real-time voice/avatar orchestration and streaming UX
- Yemaya: polished asset generation and prerecorded media workflows
- Themis: academic integrity, copyright/originality, moderation, policy
- Oshun shell/admin: identity, routing, cross-domain discovery, ops
7.3 Shared data contracts to add or formalize#
Recommended canonical shared objects:
LearningSourceBundleGroundingPackCourseBuildLessonAssetBundleTutorPersonaProfileTutorSessionMemoryLearnerMasteryStateLearningObjectiveMapAssessmentEvidencePackAvatarTeacherProfileVoiceLikenessConsentRecordLearningTelemetryStatementPublicationPackageAcademicIntegrityVerdict
7.4 Operator and governance integration#
Metis needs first-class review surfaces for:
- source/material approval
- copyright/licensing review
- curriculum alignment review
- quality/linter verdict review
- tutor persona approval
- avatar/voice consent review
- publication approval
- post-release learner-safety and integrity monitoring
7.5 Cross-domain product integration#
Minimum cross-domain integrations that should become explicit V1 work:
- Metis x Sophia: grounded course and tutor answers from approved sources
- Metis x Isis: prompt/model/version/cost governance
- Metis x Lilith: tutor pedagogy, motivational tone, coaching strategies
- Metis x Iris: learner continuity and memory
- Metis x Psyche: real-time voice/avatar tutors
- Metis x Yemaya: polished lectures, diagrams, narrated assets
- Metis x Themis: originality, rights, academic integrity, appeals
- Metis x Aja: embodied instruction and movement coaching
- Metis x Themis domain knowledge: civic/governance learning already has a bridge and should be treated as a pattern, not an isolated exception
8. Recommended Target Architecture#
8.1 Canonical learning pipeline#
BYOM source intake ->
Sophia parsing, citation graph, source quality, grounding ->
Metis course planning and assessment planning ->
Isis model/prompt routing and generation controls ->
Metis quality gates and evidence bundling -> Yemaya/Psyche media rendering
-> web/mobile/LMS delivery -> Metis tutoring loop ->
Iris memory + mastery state -> analytics/evals/retraining decisions
8.2 Tutor runtime#
Recommended tutor runtime layers:
-
Policy layer
- age/safety/integrity constraints
- subject/domain guardrails
- allowed assistance mode
-
Grounding layer
- course bundle
- approved sources
- current learner state
- prior attempts and misconceptions
-
Pedagogy layer
- Socratic mode
- direct explanation mode
- scaffolded hint mode
- exam-practice mode
- mentor/coaching mode
-
Delivery layer
- text
- audio
- avatar
- diagrams / whiteboard / code
8.3 Avatar teacher architecture#
Recommended split:
- Brain: Metis + Sophia + Lilith + Iris
- Voice: low-latency TTS stack with pronunciation/emotion controls
- Face/body renderer: Psyche-backed renderer with optional external avatar providers for production modes
- Asset mode: prerecorded lectures for scale and LMS export
- Conversation mode: real-time avatar for tutoring and office hours
8.4 BYOM architecture#
Minimum BYOM flow:
- ingest file, URL, feed, or corpus package
- extract text, structure, metadata, citations, rights, version hash
- create a bounded source bundle
- generate grounded outline and evidence map
- bind every lesson/quiz/tutor answer to allowed source scope
- preserve provenance into exports and admin review
- refresh downstream derivatives when source bundles change
9. Expert Computer Science Review of Metis Systems and Algorithms#
9.1 High-level verdict#
I reviewed the report again for completeness and correctness against the current repo, with a stricter standard:
- not whether Metis has broad scope
- but whether the implemented Metis systems can honestly be described as state-of-the-art, industry-leading, robust, and guaranteed to produce absolutely accurate educational outputs
From that stricter expert-computer-science perspective, the answer is:
- Metis has a broad and unusually ambitious foundation
- several algorithmic areas are real and technically respectable
- some planning documents materially overstate current readiness
- the current implementation is not yet state-of-the-art end-to-end
- it would be technically incorrect to describe current Metis as guaranteed or absolutely accurate
The strongest current areas are:
- classical psychometrics and adaptive assessment foundations
- typed modular domain libraries
- quality/linter architecture direction
- experimentation, cost, and model-governance scaffolding
The weakest current areas are:
- live grounded course generation
- runtime tutoring intelligence
- research/BYOM execution
- knowledge-graph extraction quality
- natural avatar educator realism
- canonical contracts, service tests, and end-to-end verification
9.2 Surface completeness and runtime truthfulness#
The report is now more explicit about a core truth:
apps/metis/webis still largely mock-backedapps/metis/adminis still largely mock-backedapps/metis/api-gatewayandapps/metis/workerare more real and better testedservices/metisis a real backend scaffold, but its tutoring and verification depth are still materially behind the architectural ambition expressed in the docs- generated Metis domain docs still include phase-derived or marketing-forward capability framing that can outrun verified runtime evidence
Inference:
Current Metis should be described as a real platform foundation with an incomplete production surface, not as a finished integrated educational system.
9.3 Agents, prompting, and orchestration#
What is strong:
libs/metis/agents/*andlibs/metis/prompt-management/*provide a coherent type system and reusable orchestration abstractions- the prompt pipeline supports multi-stage flows, retries, branching, and validation hooks
- the Minerva analysis documents preserve a strong conceptual multi-agent architecture
What is weaker than the planning documents imply:
ResearchAgentcurrently generates synthetic-looking sources, sections, and findings rather than performing verified live researchContentAgentrelies heavily on templates, readability formulas, and heuristic quality scoringTutorAgentuses keyword-driven understanding detection, template-based Socratic prompts, and heuristic difficulty adjustment- no verified runtime agent registry, async event bus, durable tool-grounded multi-agent execution layer, or production-grade debate/adjudication runtime is visible in Oshun-local Metis
Expert verdict:
Metis currently has a useful agent abstraction layer, but it does not yet have a frontier-grade agent runtime. It is closer to a strong typed simulation and orchestration substrate than to a proven state-of-the-art autonomous educational agent system.
9.4 Research, BYOM, corpus, and search#
What is strong:
libs/metis/research/*covers corpus metadata, embeddings, citations, credibility, knowledge-graph types, analytics, and search contracts- the Minerva analysis preserves a strong research-system vision with monitoring and fact-verification ambitions
What is incomplete:
corpus.tsandsearch.tsare mostly type/model/query utility layers, not a fully verified production retrieval stack- no canonical Oshun-level
LearningSourceBundleor rights/provenance release object exists yet - no verified runtime document hashing, rights lineage, or downstream source invalidation chain is visible end-to-end
- no verified hybrid retriever with claim extraction, query rewriting, and evidence ranking is shown operating across the learner/admin/runtime stack
- no verified ingestion-time or retrieval-time defense chain is shown for prompt injection, hidden instructions, malicious documents, or source poisoning in Metis BYOM paths
Expert verdict:
Metis has the shape of a serious BYOM system, but not yet the verified runtime behavior of NotebookLM-style or source-first study systems. It is currently a design-rich substrate, not a completed source-grounded learning product.
9.5 Knowledge graph and retrieval algorithms#
This is one of the areas where the new review is stricter.
Verified implementation evidence:
GraphStoreis an in-memory graph store- concept extraction uses stop-word filtering, regex patterns, n-grams, TF/IDF, Jaccard, and character n-gram similarity
- relationship extraction uses regex patterns, co-occurrence windows, and PMI-like confidence scoring
- concept similarity combines path-based, Wu-Palmer, Lin, Resnik, feature-overlap, and a simple feature-vector cosine score
- research-layer graph models are rich and well typed
What that means technically:
- these are competent classical algorithms
- they are useful baselines for educational ontology construction
- they are not the same thing as state-of-the-art entity linking, relation extraction, graph retrieval, or graph-grounded reasoning
Compared with current frontier patterns such as Microsoft GraphRAG:
- Metis does not yet show a verified hierarchical graph-indexing/query system
- Metis does not yet show community-summary or graph-cluster views that support GraphRAG-style global reasoning over large corpora
- Metis does not yet show query-time global/local/drift graph retrieval over a production corpus
- Metis does not yet show claim-level grounding tied to graph evidence in the user-facing runtime
Expert verdict:
Metis knowledge-graph code is a solid classical educational KG toolkit. It is not yet an industry-leading graph-grounded reasoning system.
9.6 Assessment, psychometrics, and adaptive learning#
This is the strongest current algorithmic area.
Verified strengths:
- real 3PL IRT probability and information calculations
- MLE and EAP-style ability estimation
- test information, standard error, and reliability utilities
- adaptive item selection with information maximization
- Sympson-Hetter exposure control
- mastery thresholds, decay modeling, and prerequisite validation
- adaptive learning utilities for ZPD-style adjustment and spaced repetition
- a simplified DKT-like knowledge tracer with transfer-learning propagation and retention prediction
Why this matters:
- these are not superficial demos
- they reflect legitimate educational measurement and adaptive-testing theory
- they are materially stronger than typical "AI tutor" products that only wrap a chatbot around generic multiple-choice generation
What still keeps this from being frontier:
- the knowledge tracer is explicitly simplified, not a modern transformer-based or encoder-decoder knowledge tracing model
- no real item-bank calibration evidence is shown
- no differential item functioning or fairness analysis is shown
- no empirical learner-outcome benchmark demonstrates superiority over simpler baselines
- no verified coupling exists between psychometric state and the live tutoring service runtime
- no verified blueprint assembly, anchor-item strategy, equating, or form-governance runtime is shown for high-stakes Metis assessments
- no verified secure-delivery or exam-integrity layer is shown for institution-facing or high-stakes assessment modes
Expert verdict:
Metis assessment and adaptive-testing foundations are genuinely strong and closer to industry-grade than most other Metis subsystems. This is the best candidate area to make industry-leading first, provided it is calibrated on real learner data and tied into production tutor behavior.
9.7 Tutoring, learner modeling, memory, and pedagogy#
What is strong:
libs/metis/tutoring/*provides good conversation, persona, hints, scaffolding, learning-style, and session abstractionslibs/metis/learning/*andlibs/metis/adaptive/*preserve a broad learner modeling vocabulary- the repo has a meaningful conceptual distinction between tutoring, assessment, mastery, and adaptation
What is currently not strong enough:
- the Python tutoring service is still a generic prompt wrapper around model providers
- no durable episodic learner memory or misconception graph is visible in the verified runtime
- no true source-scoped answer policy is enforced in the current tutor service
- no verified teacher visibility, intervention, or escalation loop is wired into the live tutoring path
- tutor persona logic exists mostly as typed structures and helper functions, not as a fully governed production pedagogy engine
Expert verdict:
Metis tutoring is architecturally promising but runtime-incomplete. It is not yet a state-of-the-art tutor from either an edtech or computer-science evaluation standpoint.
9.8 Quality control, factuality, and verification#
What is strong:
- Minerva-origin linter architecture is broad and well thought out
- the TypeScript quality scorer and linter infrastructure are operationally useful
- the codebase preserves an explicit culture of evaluation, scoring, and review
What is weaker than a strong accuracy claim would require:
- current quality scoring is largely heuristic and weighted, not a calibrated factuality guarantee mechanism
math-validator.tschecks syntax, balance, notation, and unit patterns, but is not equivalent to full symbolic proof or theorem verification in the TS runtime- the LLM client quality score is based on completeness, readability, structure, relevance, and Bloom-alignment heuristics, not direct truth verification
- no verified claim-extraction plus evidence-adjudication pipeline is enforcing correctness before publication
- no calibrated abstention/escalation policy is shown for uncertain or contradictory answers
- no adversarial evaluation suite is shown for prompt injection, source poisoning, cheating assistance, or unsafe tutor steering
- no governed gold-set, holdout-set, or calibrated human-rater workflow is shown for pedagogical and factuality evaluation
Expert verdict:
Metis has a good QA architecture direction, but it does not yet have an accuracy-assurance system strong enough to justify claims of guaranteed correctness.
9.9 Multimedia, voice, video, and avatar systems#
What is strong:
libs/metis/multimedia/*covers audio, diagrams, presentation, video, and avatar abstractions with meaningful test coverage- Minerva analysis documents show unusually rich multimedia ambition for an educational platform
- prerecorded export modes such as SCORM and packaged content are more realistic near-term strengths than full interactive avatar tutoring
What is weaker than the phrase "seamless natural avatar teachers" implies:
- the avatar generator uses text-keyword gesture mapping and simple procedural frame generation
- this is not equivalent to a production-grade natural avatar rendering stack
- no verified real-time speech-to-speech tutor stack is integrated into Metis end-to-end today
- no verified low-latency turn-taking, barge-in, speech alignment, or likeness governance stack is shown inside current Metis runtime paths
Expert verdict:
Metis multimedia is a broad abstraction layer with strong design intent. It is not yet a natural-avatar educator system at the level implied by Tavus, HeyGen, OpenAI Realtime, or Synthesia-class production experiences.
9.10 Model governance, experimentation, cost, and optimization layers#
What is strong:
model-registry,ab-testing,cost-tracking, andprompt-managementare serious operational building blocks- the A/B testing library includes real statistical testing primitives
- cost tracking and budget management are more sophisticated than many internal AI prototypes ever reach
What is incomplete:
- these subsystems are not yet visibly tied to canonical release gates for tutor quality, groundedness, or pedagogy
- no verified continuous evaluation loop is choosing or rolling back models based on real learner outcomes
- no evidence shows prompt/model experiments coupled to a production accuracy benchmark suite
- no verified shadow, canary, or champion-challenger rollout path is visible for high-stakes tutor, build, or retrieval policy changes
Expert verdict:
This is a strong MLOps-style substrate for Metis, but it is not yet the closed-loop optimization system that would make Metis industry-leading.
9.11 Infrastructure, contracts, and testing#
This area now has a stronger negative conclusion than the first report draft.
Verified facts:
- 211 test/spec files exist across
apps/metis,libs/metis, andservices/metis - no Python test files were discovered under
services/metis - no Playwright/E2E coverage was discovered for the Metis web or admin apps
- no canonical OpenAPI root for Metis was discovered, despite real service schemas and API routes
- generated Metis domain reference docs still derive major capability framing from historical phase inventories rather than current runtime verification
Expert verdict:
No expert should call Metis a truly robust, well-built, industry-leading system while those contract and verification gaps remain open. The libraries are broad; the release discipline is not yet broad enough.
9.12 Expert subsystem verdict matrix#
| Subsystem | Repo truth | Expert verdict |
|---|---|---|
| Web learner surface | Mostly mock-backed | Incomplete |
| Admin surface | Mostly mock-backed | Incomplete |
| API gateway / worker | Real and tested | Strong foundation |
| Python backend service | Real scaffold, limited test evidence | Partial |
| Agents/orchestration | Strong abstractions, weak live execution proof | Partial |
| BYOM/research runtime | Rich models, weak end-to-end runtime proof | Partial |
| Knowledge graph | Classical algorithms, not frontier retrieval | Moderate baseline |
| Assessment / IRT / CAT | Real and technically strong | Strong |
| Adaptive learning | Good classical methods, limited empirical proof | Moderate-to-strong |
| Tutoring runtime | Thin live intelligence path | Incomplete |
| Quality / factuality | Good architecture, weak guarantee surface | Partial |
| Multimedia / avatars | Broad abstractions, limited natural-runtime proof | Partial |
| Model governance / A-B / cost | Strong substrate | Strong foundation |
| Contracts / E2E / service verification | Missing critical pieces | Incomplete |
10. Accuracy, Robustness, and Assurance Standard#
10.1 Why "guaranteed absolute accuracy" is the wrong claim#
No honest expert review should promise that a modern AI educational platform can guarantee absolute accuracy across all generated content, tutor responses, knowledge-graph edges, and avatar-mediated explanations.
Why:
- model outputs are probabilistic
- retrieval systems can miss or mis-rank evidence
- OCR and document parsing can introduce source errors
- knowledge graphs can encode extraction mistakes or stale facts
- multi-agent pipelines compound uncertainty unless carefully audited
- ambiguity in user prompts and source materials can create multiple plausible interpretations
Inference:
The right target is not a marketing-style guarantee. The right target is a high-assurance system with explicit error bounds, provenance, abstention, and escalation.
10.2 What Metis should guarantee instead#
Metis should guarantee:
- bounded source scope for grounded tasks
- deterministic versioned build artifacts and hashes
- claim-to-source traceability for high-stakes outputs
- explicit surfacing of missing evidence and low-confidence regions
- calibrated abstention when confidence is below threshold
- human-review escalation for high-stakes or contradictory cases
- durable audit logs for publication, tutoring, and moderation decisions
- screened source-ingestion and retrieval paths when external materials can carry prompt-injection, hidden-instruction, or poisoning risk
- versioned benchmark, rater, and rollout evidence for model, prompt, and retrieval policy changes
10.3 Required assurance architecture#
For Metis to become truly robust, every publishable course build, assessment
build, and high-stakes tutor answer path should pass through an EvidenceBundle
pipeline with at least these stages:
-
Source integrity
- file hash
- source identity
- rights/licensing record
- parser/OCR confidence
-
Source threat screening
- prompt-injection and hidden-instruction detection
- malicious document and unsafe-link screening
- source-poisoning checks and trust-zone classification
-
Claim extraction
- extract atomic claims, formulas, definitions, dates, entities, and prerequisite assertions
-
Hybrid retrieval
- lexical retrieval
- dense retrieval
- graph-aware retrieval
- source diversification
-
Verifier ensemble
- citation checker
- contradiction checker
- symbolic/domain validators for math, code, units, and structured facts
- style/pedagogy linters
-
Confidence calibration
- per-claim confidence
- per-artifact confidence
- uncertainty thresholds for abstain/escalate behavior
-
Human review gate
- required for policy-sensitive, low-confidence, or high-impact outputs
10.4 State-of-the-art target definition for Metis#
From an expert-computer-science standpoint, Metis should only be called state-of-the-art when it can demonstrate all of the following:
- source-grounded generation by default for high-stakes outputs
- graph- and retrieval-aware reasoning rather than prompt-only synthesis
- empirically validated adaptive learning models against modern baselines
- low-latency multimodal tutoring with measurable turn-taking quality
- calibrated factuality and pedagogical evaluation gates
- release-time and post-release monitoring tied to rollback decisions
- adversarially tested BYOM and retrieval paths that resist prompt injection and source poisoning
- governed benchmark sets and calibrated human-rater evaluation for pedagogy and factuality
- canonical documentation and scorecards that stay aligned with verified runtime truth
11. Concrete Recommendations#
11.1 P0: Canonical correction and product baseline#
- Add this report to the canonical OSHUN V1 derived-source set.
- Add an explicit Metis program workstream to
docs/releases/v1/specs/todos.md. - Add
metisto all Oshun shell/domain registry contracts wherenisabahad to be added. - Define Metis as both a first-class domain and a shared educational backbone.
- Create a canonical Metis V1 scorecard for learner-facing, admin-facing, platform-facing, and governance-facing completeness.
11.2 P0: Turn Metis from mock product into live product#
- Replace
apps/metis/webmock course, auth, tutoring, progress, and creation flows with live API-backed flows. - Replace
apps/metis/adminmock moderation, approval, monitoring, and user management flows with live API-backed flows. - Add Playwright coverage for
apps/metis/webandapps/metis/admin. - Add Python backend tests for
services/metis. - Create or publish the missing Metis OpenAPI contract surface.
11.3 P0: Grounded content generation#
- Formalize
LearningSourceBundle,GroundingPack, andCourseBuildcontracts. - Route all high-stakes content generation through Sophia-backed source grounding and citation verification.
- Require provenance bundles for published courses, assessments, and tutor answer paths.
- Add operator review for source rights, source quality, and generation release approval.
11.4 P0: Real tutor, not just chat completion#
- Replace the current generic tutoring prompt path with a tutoring orchestrator that reads learner state, course context, prior attempts, and source scope.
- Add durable learner memory via Iris.
- Add mastery and misconception state updates after every session and assessment attempt.
- Support at least four tutor modes: Socratic, direct explanation, scaffolded hints, and exam-practice.
- Add tutor integrity modes that distinguish "teach me" from "do this for me."
11.5 P1: BYOM as a first-class product#
- Ship notebook/workspace scoped source ingestion with course-author and learner-facing source visibility.
- Support versioned source refresh and downstream invalidation of generated derivatives.
- Add connectors/importers for files, web sources, feeds, and LMS content packs.
- Add source-aware study guide, flashcard, quiz, and tutor-generation flows.
11.6 P1: Natural avatar educators#
- Ship prerecorded avatar lecturers for polished lessons and LMS exports.
- Ship real-time voice tutors before shipping full real-time avatar tutors.
- Add explicit
AvatarTeacherProfileandVoiceLikenessConsentRecordcontracts. - Separate latency-sensitive real-time avatar tutoring from batch-rendered lecture production.
- Add multilingual narration, dubbing, captions, transcripts, and quality gates.
11.7 P1: Governance and safety#
- Add academic-integrity rules, plagiarism/originality checks, and assistance-mode disclosure.
- Add learner-safety policies for minors, teacher visibility controls, and escalation paths.
- Add avatar/voice misuse protections and consent logging.
- Add copyright/licensing review for BYOM and generated derivatives.
11.8 P1: Standards and analytics#
- Treat LTI 1.3 / LTI Advantage as the baseline LMS integration target.
- Treat xAPI/cmi5-compatible telemetry as the canonical learning analytics direction.
- Tie tutor actions, learner state changes, assessments, and media playback into a shared evaluation and experimentation layer.
11.9 P0/P1: Proof obligations for a truly robust system#
- Create a claim-level
EvidenceBundlefor every published lesson, assessment, and high-stakes tutor answer path. - Benchmark Metis adaptive models against modern knowledge-tracing baselines rather than only against legacy heuristics.
- Add calibrated confidence scoring and abstention policies for uncertain tutor and generation outputs.
- Wire model registry, experimentation, and rollback policies to real factual, pedagogical, and safety evaluation metrics.
- Route STEM-heavy outputs through symbolic or numerical verification wherever feasible instead of relying only on LLM self-consistency.
11.10 P0/P1: Adversarial hardening, evaluation governance, and runtime truth#
- Audit and correct Metis domain reference docs, feature inventories, implementation maps, and generated scorecards so they reflect verified runtime truth rather than historical phase claims.
- Add automated documentation-evidence checks so Metis capability claims cannot outrun test evidence, API evidence, or live-surface status.
- Add BYOM threat screening and retrieval-time defenses for prompt injection, hidden instructions, unsafe links, and source poisoning.
- Add adversarial evaluation suites for prompt injection, cheating assistance, policy evasion, and unsafe tutor steering.
- Create versioned gold sets, holdout sets, annotation rubrics, and calibrated human-rating workflows for groundedness, pedagogy, and tutor quality evaluation.
- Add privacy-preserving de-identification and approval controls for learner data used in calibration, evaluation, or experimentation.
- Add shadow, canary, and champion-challenger rollout modes for tutor, build, and retrieval policy changes.
- Add hierarchical graph indexing, community summaries, and query-time global/local retrieval where graph-backed reasoning is a release goal.
- Run symbolic, code, and numeric verification tools in auditable sandboxes with quotas, traceability, and failure isolation.
11.11 P1: Institutional interoperability, assessment portability, and#
learner-record depth
- Add OneRoster-grade rostering, resource, and gradebook interoperability where Metis institutional delivery is promised.
- Add institutional federation and provisioning support through SSO, SCIM, or equivalent identity-lifecycle patterns where Metis is deployed into managed programs.
- Map
LearningObjectiveMapand competency evidence to CASE-compatible standards frameworks and preserve objective lineage end to end. - Add QTI 3 and CAT-aligned assessment import/export and conformance testing where Metis promises portable or adaptive institutional assessments.
- Add Caliper-compatible event emission or bridge layers where institutional analytics ecosystems require it in addition to xAPI/cmi5.
- Add CLR / Open Badges-compatible learner-record and achievement export where Metis promises portable records, badges, or credentials.
- Add production assessment blueprinting, item-bank governance, anchor-item strategy, equating, and exposure-control operations for high-stakes or adaptive Metis assessments.
- Add secure-delivery and exam-integrity modes, including proctoring or secure-browser hooks where such modes are promised.
- Add multilingual content, tutor, and assessment evaluation across supported locales plus post-release drift monitoring by subject, grade band, institution, and language.
12. Recommended Success Criteria#
Metis should not be considered fully integrated into OSHUN V1 until all of the following are true:
metisexists in the canonical Oshun shell/domain model- learner web and admin surfaces are API-backed, not mock-backed
- grounded BYOM flow exists end-to-end
- tutoring is source-aware, mastery-aware, and memory-aware
- prerecorded lecture generation and real-time tutoring are both operational
- avatar/voice modes have explicit safety, consent, and fallback policies
- Metis is represented in canonical admin/review/governance surfaces
- Playwright coverage exists for critical learner/admin flows
- Python backend tests exist for service-critical paths
- release gates include quality, rights, safety, cost, and pedagogy checks
- release gates emit claim-level provenance and confidence evidence
- adaptive systems have empirical benchmark evidence against modern baselines
- high-stakes outputs have deterministic audit trails and escalation thresholds
- BYOM and agentic retrieval paths are tested against prompt injection, malicious-source, and source-poisoning cases
- benchmark suites use governed gold/holdout datasets and calibrated human review
- Metis domain docs and launch scorecards remain aligned with verified runtime truth
- high-stakes verification tools execute in isolated sandboxes with audit trails
- institutional interoperability promises such as OneRoster, QTI, CASE, CLR, Open Badges, or Caliper are only exposed where they are implemented, tested, and operationally supported
- supported Metis locales have content, tutor, and assessment quality evidence rather than English-only assumptions
- high-stakes and adaptive Metis assessments use blueprinting, exposure control, and integrity controls where such modes are promised
- post-release Metis dashboards detect groundedness, pedagogy, fairness, mastery, and integrity drift by relevant cohorts
13. Immediate Backlog Seeds#
These are the first 15 backlog items I would import into the canonical V1 execution backlog immediately:
- METIS-V1-001 Add
metisto canonical Oshun V1 domain scope, scorecards, and release gates. - METIS-V1-002 Add
metisto shell navigation, domain registry, analytics, notifications, assistant context, and deep-link contracts. - METIS-V1-003 Replace Metis web mock auth/data/tutoring flows with live API-backed flows.
- METIS-V1-004 Replace Metis admin mock dashboards/workflows with live API-backed flows.
- METIS-V1-005 Publish canonical Metis OpenAPI contract and align gateway, web, admin, and backend to it.
- METIS-V1-006 Create Playwright coverage for learner browse, course play, tutoring, assessment, and course-creation flows.
- METIS-V1-007 Create Playwright coverage for admin approvals, moderation, monitoring, and user-management flows.
- METIS-V1-008 Add pytest coverage for Metis tutoring, assessment, course, and analytics service paths.
- METIS-V1-009 Formalize
LearningSourceBundle,GroundingPack, andCourseBuildcontracts. - METIS-V1-010 Route course generation and tutor answers through Sophia-backed grounding and provenance.
- METIS-V1-011 Rebuild tutoring around learner memory, mastery state, and source-aware response policies.
- METIS-V1-012 Define prerecorded lecturer mode, live voice tutor mode, and live avatar tutor mode as separate product/runtime tiers.
- METIS-V1-013 Create a claim-level
EvidenceBundlepipeline with verifier ensemble outputs and audit logs. - METIS-V1-014 Benchmark knowledge tracing, adaptive selection, and mastery models against modern baselines and calibrate on real learner data.
- METIS-V1-015 Tie model registry, A/B testing, and rollout decisions to factuality, pedagogy, latency, safety, and cost evaluation gates.
14. External Sources#
Official sources accessed on March 23, 2026:
- OpenAI ChatGPT Enterprise & Edu release notes: https://help.openai.com/en/articles/10128477-chatgpt-enterprise-edu-release-notes
- OpenAI Realtime WebRTC guide: https://developers.openai.com/api/docs/guides/realtime-webrtc
- OpenAI Realtime VAD guide: https://developers.openai.com/api/docs/guides/realtime-vad
- Google Gemini study materials help: https://support.google.com/gemini/answer/16275879
- Google NotebookLM notebook/source help: https://support.google.com/notebooklm/answer/16206563
- Khan Academy Khanmigo teacher tools help: https://support.khanacademy.org/hc/en-us/articles/28494037644941-Khanmigo-Teacher-Tools-now-available-in-English-to-locales-outside-the-United-States
- Khan Academy Khanmigo community guidelines: https://support.khanacademy.org/hc/en-us/articles/13860282793869-What-are-the-Community-Guidelines-for-Khanmigo
- Duolingo Max / Video Call with Lily: https://blog.duolingo.com/duolingo-max/
- Tavus conversational video docs: https://docs.tavus.io/sections/conversational-video-interface/echo-mode
- HeyGen LiveAvatar help: https://help.heygen.com/en/articles/12758516-introducing-liveavatar
- ElevenLabs text-to-speech API: https://elevenlabs.io/text-to-speech-api
- Synthesia product page: https://www.synthesia.io/
- 1EdTech LTI security and LTI 1.3 recommendation: https://www.1edtech.org/lti-security-announcement-and-deprecation-schedule
- 1EdTech OneRoster standard: https://www.1edtech.org/standards/oneroster
- 1EdTech QTI standard: https://www.1edtech.org/standards/qti
- 1EdTech CAT standard: https://www.1edtech.org/standards/cat
- 1EdTech Caliper standard: https://www.1edtech.org/standards/caliper
- 1EdTech CLR standard: https://www.1edtech.org/standards/clr
- 1EdTech Open Badges standard: https://www.1edtech.org/standards/open-badges
- 1EdTech CASE ecosystem: https://www.1edtech.org/case-global-ecosystem
- Microsoft GraphRAG documentation: https://microsoft.github.io/graphrag/
- Microsoft GraphRAG GitHub: https://github.com/microsoft/graphrag
- OWASP Prompt Injection: https://owasp.org/www-community/attacks/PromptInjection
- ADL Experience API (xAPI) project: https://www.adlnet.gov/projects/xapi/
- Self-RAG paper: https://arxiv.org/abs/2310.11511
- SAINT knowledge tracing paper: https://arxiv.org/abs/2002.07033
15. Final Assessment#
Metis should now be treated as a missing canonical V1 domain plus a shared educational platform layer.
The repo already contains enough Metis foundation that excluding it from OSHUN V1 planning is no longer defensible.
The correct next step is not a minor documentation tweak. The correct next step is to make Metis part of the canonical V1 program, then execute the grounded content, tutor, BYOM, and avatar work needed to bring it to a genuinely state-of-the-art level.
The revised expert conclusion is narrower and more defensible:
- Metis is already a serious educational-platform foundation
- Metis is not yet an honestly state-of-the-art end-to-end educational system
- Metis can plausibly become industry-leading, but only if it earns that status through source grounding, empirical evaluation, calibrated assurance, and production verification rather than through checklist completion claims