Status: Active
Date: April 10, 2026
Owners: OSHUN Engineering, OSHUN Platform, OSHUN Safety
Related:
docs/releases/v1/specs/todos.mddocs/releases/v1/reports/metis-v1-scorecard.mddocs/releases/v1/reports/metis-v1-workstream-tracking.mddocs/releases/v1/reports/metis-subsystem-ownership-matrix.mddocs/releases/v1/specs/shared-object-persistence-migration-backfill-plan.mddocs/releases/v1/specs/metis-learning-source-bundle-contract.mddocs/releases/v1/specs/metis-grounding-pack-contract.mddocs/releases/v1/specs/metis-course-build-contract.mddocs/releases/v1/specs/metis-publication-package-contract.mddocs/releases/v1/specs/metis-voice-likeness-consent-record-contract.mdservices/metis/src/metis/services/source_ingestion_service.pyservices/metis/src/metis/services/learner_mastery_service.pyservices/metis/src/metis/services/learner_continuity_service.pyservices/metis/src/metis/services/lecture_generation_service.pyservices/metis/src/metis/services/tutoring_service.pyservices/metis/src/metis/tasks/export.py
1. Purpose#
This document is the authoritative plan for V1-MET-104: define persistence,
migration, and backfill plans for the new Metis source-bundle, mastery,
tutor-memory, build-record, and consent-record object families.
It exists because Metis now has canonical contracts and live service logic for those families, but the current repo still spans three different durability patterns:
- SQLAlchemy-backed relational rows for courses, assessments, enrollments, submissions, tutoring sessions, and tutoring messages
- file-backed JSON manifests rooted under
METIS_SOURCE_INGESTION_STORAGE_DIR - metadata blobs embedded inside existing session, message, or export payloads
This plan defines how those precursor stores are retired without allowing ownership drift away from the subsystem matrix.
2. Existing Durable Patterns To Reuse#
Metis should reuse the repo's existing migration and backfill patterns rather than invent a Metis-only format.
| Existing implementation | Why it matters |
|---|---|
docs/releases/v1/specs/shared-object-persistence-migration-backfill-plan.md |
It is the canonical Oshun-wide migration and cutover template. |
libs/contracts/src/common/contract-prisma-migrations.test.ts |
It establishes the expected migration proof standard for owner-bound contract persistence. |
apps/isis/output-registry/src/services/output.service.ts |
It demonstrates idempotent backfill, marker-based replay, and provenance-safe enrichment. |
services/metis/src/metis/services/source_ingestion_service.py |
It is the current raw source-package durability layer and the highest-fidelity precursor for LearningSourceBundle migration. |
services/metis/src/metis/services/learner_mastery_service.py |
It is the current mastery snapshot writer and the precursor source for canonical learner mastery persistence. |
services/metis/src/metis/services/learner_continuity_service.py |
It is the current continuity-memory writer and the precursor source for Iris-owned tutor/session memory migration. |
services/metis/src/metis/services/lecture_generation_service.py |
It is the current durable lecture/build package writer and the precursor source for Metis build and publication records. |
services/metis/src/metis/services/tutoring_service.py |
It contains the current consent-bearing tutoring policy, handoff, runtime-fallback, and representation metadata that must be split. |
services/metis/src/metis/tasks/export.py |
It shows the current export-grade package assembly and the build/export provenance surface that backfill verification must preserve. |
3. Authoritative Owner Map#
The owner map is already settled by
docs/releases/v1/reports/metis-subsystem-ownership-matrix.md.
| Object family | Canonical owner | Canonical durable target | Current precursor sources |
|---|---|---|---|
LearningSourceBundle |
Sophia | Sophia-owned source-bundle rows plus migration ledger and source-to-bundle crosswalk | source_ingestion_service.py, extraction/refresh/outline/generation/concept graph manifests under file storage |
LearnerMasteryState |
Metis | Metis-owned mastery rows and event ledger keyed to learner, topic, course, and session | learner_mastery_service.py, assessment submissions, tutoring turns, analytics-derived mastery context |
TutorSessionMemory |
Iris | Iris-owned durable session-memory and continuity rows with consent-aware retention scope | learner_continuity_service.py, tutoring-session summaries, session/message metadata |
CourseBuild and build records |
Metis | Metis-owned build, lecture-package, publication, evaluation, and export package rows | lecture_generation_service.py, source generation/evaluation outputs, export task payloads |
VoiceLikenessConsentRecord and related consent records |
Themis | Themis-owned consent and approval records with Metis crosswalk references | tutoring-session teacher-representation metadata, consent logs, admin review queue artifacts |
Normative rules:
- Metis may continue to orchestrate these objects, but it must not become the durable owner for Sophia, Iris, or Themis records.
- File-backed manifests under
METIS_SOURCE_INGESTION_STORAGE_DIRare a precursor store, not an acceptable long-term system of record. - Embedded metadata JSON in tutoring messages is a compatibility seam, not the final durable model for memory or consent state.
4. Current Precursor Storage Surfaces#
4.1 File-backed precursor stores#
The current file-backed state under METIS_SOURCE_INGESTION_STORAGE_DIR
contains the highest-fidelity precursor data for several new object families:
- source package manifests and artifacts
- extraction results
- review state and usage-policy state
- grounded outlines
- grounded generation artifacts
- retrieval and concept-graph artifacts
- study-workspace projections
- learner mastery snapshots in
learner-mastery/*.json - learner continuity memory in
learner-continuity/*.json - lecture packages in
lecture-packages/<course-id>/latest.json
4.2 Relational precursor stores#
Metis already has relational rows for:
courses,modules,lessonsassessments,questions,submissionsenrollments,progress,achievementstutoring_sessions,tutoring_messagesusers.preferences
These rows already carry source IDs, topic labels, progress markers, lesson links, grading outputs, and tutoring metadata needed to backfill the new object families.
4.3 Embedded metadata seams to retire#
The following data is currently embedded and must be lifted into explicit records or crosswalks:
- tutoring delivery-fallback policy
- oversight and integrity policy
- teacher representation and live runtime metadata
- consent logs and consent summaries
- export package provenance summaries when only embedded in export payloads
5. Cross-Cutting Migration Rules#
5.1 ID and crosswalk strategy#
- Preserve stable IDs where the current contract already exposes them and the owning subsystem accepts them.
- Otherwise generate deterministic UUIDv5 values from:
metis + object_family + precursor_store + precursor_id. - Persist a migration crosswalk for every migrated row: precursor store, precursor ID, canonical owner, canonical ID, content hash, migrated at, and latest verification status.
5.2 Replay and idempotency#
- Every backfill runner must be upsert-based.
- Every runner must maintain a migration ledger with: object family, source cursor, canonical owner, canonical ID, source hash, started at, finished at, status, and last error.
- Re-running the same migration with the same source hash must be a no-op.
- A changed source hash must create either an in-place update or a new versioned canonical row according to the target contract's versioning rules.
5.3 Cutover stages#
Each Metis object family follows the same stages:
- Define target schema and migration ledger.
- Run dry-run extraction and contract-shape validation.
- Backfill from file-backed or embedded precursor sources.
- Enable dual-write where the old write path is still active.
- Run parity checks on counts, hashes, and representative samples.
- Switch reads to the canonical owner or owner-backed adapter.
- Freeze and then retire the legacy precursor path.
5.4 What must not be backfilled#
- transient tutoring websocket/runtime tokens
- shell-only continuation cards
- in-browser derived analytics summaries
- learner notification projections
- grounded inspector view state
- admin dashboard summary tiles
6. Object-Specific Plans#
6.1 LearningSourceBundle migration#
Canonical owner:
- Sophia owns the durable bundle and review lineage.
Primary precursor sources:
SourceIngestionServicepackage manifests- extraction manifests
- review and usage-policy state
- refresh/version-diff manifests
- connector previews, concept graphs, outlines, and generation manifests
Required backfill sequence:
- Register a source-bundle migration ledger and scope crosswalk.
- Import raw package shell rows first: scope, requestor, source version, item inventory, package status.
- Attach extraction and citation manifests.
- Attach review history and usage-policy state.
- Attach derived asset references: outline, generation artifacts, retrieval indices, concept graphs, refresh/version history, study-workspace references.
- Mark bundle rows as cutover-ready only after all derived artifact pointers are reproducible from the canonical row.
Special rule:
- Metis must not keep a parallel long-term copy of source trust and approval state after Sophia cutover. Metis may cache bundle summaries for learner and admin views, but those caches are rebuildable read models only.
6.2 LearnerMasteryState migration#
Canonical owner:
- Metis owns mastery semantics and mastery-state decisions.
Primary precursor sources:
learner_mastery/*.json- assessment submissions and fairness/calibration outputs
- tutoring turn updates recorded through
LearnerMasteryService - adaptive-learning and analytics-derived mastery recommendations
Required backfill sequence:
- Create a mastery ledger keyed by learner, focus topic, course, and source evidence version.
- Import the current mastery snapshot for each learner.
- Rebuild event history where available from: tutoring turns, assessment attempts, retry outcomes, and misconception links.
- Attach prerequisite and misconception graph references used in adaptive sequencing.
- Recompute and compare derived mastery context before cutover.
Parity checks:
- same weakest-skill set
- same blocked-prerequisite set
- same mastery probability bucket for representative topics
- same adaptive next-step recommendation class
6.3 TutorSessionMemory migration#
Canonical owner:
- Iris owns durable continuity and tutor memory persistence.
Primary precursor sources:
learner-continuity/*.json- tutoring session summaries and timestamps
- study-workspace session records
- topic, lesson, course, and evidence references embedded in tutoring context
Required backfill sequence:
- Create an Iris-bound memory export format with retention and consent labels.
- Backfill continuity snapshots as Iris memory rows with learner, scope, session kind, topic list, evidence summary, and preference state.
- Backfill tutoring-session links so memory rows can point back to canonical
tutoring_sessionswithout duplicating the full transcript. - Dual-write new continuity updates to the Iris owner path before read cutover.
- Remove Metis-local JSON writes only after Iris parity and deletion/export tests pass.
Special rule:
- Metis may continue to compute learner-facing continuity summaries, but it must not remain the durable memory store once Iris cutover completes.
6.4 Build-record migration#
Canonical owner:
- Metis owns
CourseBuild, lecture-package, evaluation, export-package, and publication package durability.
Primary precursor sources:
lecture-packages/<course-id>/latest.json- grounded generation and evaluation manifests
- export provenance payloads and generated package manifests
- publication/review queue artifacts that point to build packages
Required backfill sequence:
- Create canonical build rows keyed by course, source-bundle version, and build purpose.
- Import lecture packages and lesson asset bundles.
- Import grounded generation artifacts and evaluation results.
- Import export-package metadata and package manifests.
- Backfill publication-package rows and review cross-references.
- Run parity checks on lesson count, locale variants, asset manifests, provenance summaries, and review readiness state.
Special rule:
- A build record is not complete unless it can still point back to the source bundle and grounding evidence that produced it.
6.5 Consent-record migration#
Canonical owner:
- Themis owns consent and approval authority.
Primary precursor sources:
- tutoring teacher-representation policy metadata
- tutoring representation consent logs
- review queue artifacts for likeness and voice approvals
- avatar-release workflow records
Required backfill sequence:
- Extract each distinct consent-bearing representation reference from tutoring metadata and review queues.
- Backfill a Themis-owned consent record or consent crosswalk for: likeness ID, voice ID, governing artifact, status, expiry, source event, and affected runtime modes.
- Link existing tutoring sessions to the canonical consent record IDs.
- Keep a Metis compatibility projection for learner disclosure and export, but not a second durable consent truth.
Special rule:
- Blocked or expired consent states must be preserved exactly during migration; migration must never upgrade a consent decision implicitly.
7. Sequencing and Rollout Order#
The required order is:
LearningSourceBundleCourseBuildand publication/build recordsLearnerMasteryStateTutorSessionMemory- consent records and consent-to-session backfill
Why this order:
- source bundles are upstream of grounded outlines, generation, and build artifacts
- build records depend on source and grounding lineage
- mastery backfill depends on stable assessment and tutoring linkage
- tutor memory backfill depends on canonical tutoring-session references
- consent backfill must land last so it can link to final runtime and review artifact IDs rather than precursor placeholders
8. Verification and Rollback#
8.1 Verification requirements#
Each backfill wave must prove:
- row counts or object counts match the precursor source
- representative object hashes or semantic equivalence checks pass
- source-to-target crosswalk rows exist for every migrated object
- replaying the same migration is idempotent
- export and deletion paths still honor retention and consent boundaries
8.2 Rollback requirements#
- No cutover may delete precursor files or embedded metadata until the target owner passes parity checks.
- The migration ledger must be able to mark a wave as rolled back without destroying the precursor source.
- Read cutover must be reversible behind a feature flag or owner-backed adapter toggle until two clean verification runs succeed.
9. Exit Criteria For V1-MET-104#
V1-MET-104 is complete only when:
- this plan remains the canonical Metis migration/backfill reference
- the owner map aligns with the Metis subsystem matrix
- every target family has a named precursor store, cutover order, and rollback rule
- the Metis scorecard continues to point service/persistence reliability work at this migration plan plus the follow-on implementation and observability tasks