Eve library

OSHUN V1 Metis Persistence, Migration, and Backfill Plan

Spec

Revision 1

On this page

Status: Active

Date: April 10, 2026

Owners: OSHUN Engineering, OSHUN Platform, OSHUN Safety

Related:

  • docs/releases/v1/specs/todos.md
  • docs/releases/v1/reports/metis-v1-scorecard.md
  • docs/releases/v1/reports/metis-v1-workstream-tracking.md
  • docs/releases/v1/reports/metis-subsystem-ownership-matrix.md
  • docs/releases/v1/specs/shared-object-persistence-migration-backfill-plan.md
  • docs/releases/v1/specs/metis-learning-source-bundle-contract.md
  • docs/releases/v1/specs/metis-grounding-pack-contract.md
  • docs/releases/v1/specs/metis-course-build-contract.md
  • docs/releases/v1/specs/metis-publication-package-contract.md
  • docs/releases/v1/specs/metis-voice-likeness-consent-record-contract.md
  • services/metis/src/metis/services/source_ingestion_service.py
  • services/metis/src/metis/services/learner_mastery_service.py
  • services/metis/src/metis/services/learner_continuity_service.py
  • services/metis/src/metis/services/lecture_generation_service.py
  • services/metis/src/metis/services/tutoring_service.py
  • services/metis/src/metis/tasks/export.py

1. Purpose#

This document is the authoritative plan for V1-MET-104: define persistence, migration, and backfill plans for the new Metis source-bundle, mastery, tutor-memory, build-record, and consent-record object families.

It exists because Metis now has canonical contracts and live service logic for those families, but the current repo still spans three different durability patterns:

  • SQLAlchemy-backed relational rows for courses, assessments, enrollments, submissions, tutoring sessions, and tutoring messages
  • file-backed JSON manifests rooted under METIS_SOURCE_INGESTION_STORAGE_DIR
  • metadata blobs embedded inside existing session, message, or export payloads

This plan defines how those precursor stores are retired without allowing ownership drift away from the subsystem matrix.

2. Existing Durable Patterns To Reuse#

Metis should reuse the repo's existing migration and backfill patterns rather than invent a Metis-only format.

Existing implementation Why it matters
docs/releases/v1/specs/shared-object-persistence-migration-backfill-plan.md It is the canonical Oshun-wide migration and cutover template.
libs/contracts/src/common/contract-prisma-migrations.test.ts It establishes the expected migration proof standard for owner-bound contract persistence.
apps/isis/output-registry/src/services/output.service.ts It demonstrates idempotent backfill, marker-based replay, and provenance-safe enrichment.
services/metis/src/metis/services/source_ingestion_service.py It is the current raw source-package durability layer and the highest-fidelity precursor for LearningSourceBundle migration.
services/metis/src/metis/services/learner_mastery_service.py It is the current mastery snapshot writer and the precursor source for canonical learner mastery persistence.
services/metis/src/metis/services/learner_continuity_service.py It is the current continuity-memory writer and the precursor source for Iris-owned tutor/session memory migration.
services/metis/src/metis/services/lecture_generation_service.py It is the current durable lecture/build package writer and the precursor source for Metis build and publication records.
services/metis/src/metis/services/tutoring_service.py It contains the current consent-bearing tutoring policy, handoff, runtime-fallback, and representation metadata that must be split.
services/metis/src/metis/tasks/export.py It shows the current export-grade package assembly and the build/export provenance surface that backfill verification must preserve.

3. Authoritative Owner Map#

The owner map is already settled by docs/releases/v1/reports/metis-subsystem-ownership-matrix.md.

Object family Canonical owner Canonical durable target Current precursor sources
LearningSourceBundle Sophia Sophia-owned source-bundle rows plus migration ledger and source-to-bundle crosswalk source_ingestion_service.py, extraction/refresh/outline/generation/concept graph manifests under file storage
LearnerMasteryState Metis Metis-owned mastery rows and event ledger keyed to learner, topic, course, and session learner_mastery_service.py, assessment submissions, tutoring turns, analytics-derived mastery context
TutorSessionMemory Iris Iris-owned durable session-memory and continuity rows with consent-aware retention scope learner_continuity_service.py, tutoring-session summaries, session/message metadata
CourseBuild and build records Metis Metis-owned build, lecture-package, publication, evaluation, and export package rows lecture_generation_service.py, source generation/evaluation outputs, export task payloads
VoiceLikenessConsentRecord and related consent records Themis Themis-owned consent and approval records with Metis crosswalk references tutoring-session teacher-representation metadata, consent logs, admin review queue artifacts

Normative rules:

  • Metis may continue to orchestrate these objects, but it must not become the durable owner for Sophia, Iris, or Themis records.
  • File-backed manifests under METIS_SOURCE_INGESTION_STORAGE_DIR are a precursor store, not an acceptable long-term system of record.
  • Embedded metadata JSON in tutoring messages is a compatibility seam, not the final durable model for memory or consent state.

4. Current Precursor Storage Surfaces#

4.1 File-backed precursor stores#

The current file-backed state under METIS_SOURCE_INGESTION_STORAGE_DIR contains the highest-fidelity precursor data for several new object families:

  • source package manifests and artifacts
  • extraction results
  • review state and usage-policy state
  • grounded outlines
  • grounded generation artifacts
  • retrieval and concept-graph artifacts
  • study-workspace projections
  • learner mastery snapshots in learner-mastery/*.json
  • learner continuity memory in learner-continuity/*.json
  • lecture packages in lecture-packages/<course-id>/latest.json

4.2 Relational precursor stores#

Metis already has relational rows for:

  • courses, modules, lessons
  • assessments, questions, submissions
  • enrollments, progress, achievements
  • tutoring_sessions, tutoring_messages
  • users.preferences

These rows already carry source IDs, topic labels, progress markers, lesson links, grading outputs, and tutoring metadata needed to backfill the new object families.

4.3 Embedded metadata seams to retire#

The following data is currently embedded and must be lifted into explicit records or crosswalks:

  • tutoring delivery-fallback policy
  • oversight and integrity policy
  • teacher representation and live runtime metadata
  • consent logs and consent summaries
  • export package provenance summaries when only embedded in export payloads

5. Cross-Cutting Migration Rules#

5.1 ID and crosswalk strategy#

  • Preserve stable IDs where the current contract already exposes them and the owning subsystem accepts them.
  • Otherwise generate deterministic UUIDv5 values from: metis + object_family + precursor_store + precursor_id.
  • Persist a migration crosswalk for every migrated row: precursor store, precursor ID, canonical owner, canonical ID, content hash, migrated at, and latest verification status.

5.2 Replay and idempotency#

  • Every backfill runner must be upsert-based.
  • Every runner must maintain a migration ledger with: object family, source cursor, canonical owner, canonical ID, source hash, started at, finished at, status, and last error.
  • Re-running the same migration with the same source hash must be a no-op.
  • A changed source hash must create either an in-place update or a new versioned canonical row according to the target contract's versioning rules.

5.3 Cutover stages#

Each Metis object family follows the same stages:

  1. Define target schema and migration ledger.
  2. Run dry-run extraction and contract-shape validation.
  3. Backfill from file-backed or embedded precursor sources.
  4. Enable dual-write where the old write path is still active.
  5. Run parity checks on counts, hashes, and representative samples.
  6. Switch reads to the canonical owner or owner-backed adapter.
  7. Freeze and then retire the legacy precursor path.

5.4 What must not be backfilled#

  • transient tutoring websocket/runtime tokens
  • shell-only continuation cards
  • in-browser derived analytics summaries
  • learner notification projections
  • grounded inspector view state
  • admin dashboard summary tiles

6. Object-Specific Plans#

6.1 LearningSourceBundle migration#

Canonical owner:

  • Sophia owns the durable bundle and review lineage.

Primary precursor sources:

  • SourceIngestionService package manifests
  • extraction manifests
  • review and usage-policy state
  • refresh/version-diff manifests
  • connector previews, concept graphs, outlines, and generation manifests

Required backfill sequence:

  1. Register a source-bundle migration ledger and scope crosswalk.
  2. Import raw package shell rows first: scope, requestor, source version, item inventory, package status.
  3. Attach extraction and citation manifests.
  4. Attach review history and usage-policy state.
  5. Attach derived asset references: outline, generation artifacts, retrieval indices, concept graphs, refresh/version history, study-workspace references.
  6. Mark bundle rows as cutover-ready only after all derived artifact pointers are reproducible from the canonical row.

Special rule:

  • Metis must not keep a parallel long-term copy of source trust and approval state after Sophia cutover. Metis may cache bundle summaries for learner and admin views, but those caches are rebuildable read models only.

6.2 LearnerMasteryState migration#

Canonical owner:

  • Metis owns mastery semantics and mastery-state decisions.

Primary precursor sources:

  • learner_mastery/*.json
  • assessment submissions and fairness/calibration outputs
  • tutoring turn updates recorded through LearnerMasteryService
  • adaptive-learning and analytics-derived mastery recommendations

Required backfill sequence:

  1. Create a mastery ledger keyed by learner, focus topic, course, and source evidence version.
  2. Import the current mastery snapshot for each learner.
  3. Rebuild event history where available from: tutoring turns, assessment attempts, retry outcomes, and misconception links.
  4. Attach prerequisite and misconception graph references used in adaptive sequencing.
  5. Recompute and compare derived mastery context before cutover.

Parity checks:

  • same weakest-skill set
  • same blocked-prerequisite set
  • same mastery probability bucket for representative topics
  • same adaptive next-step recommendation class

6.3 TutorSessionMemory migration#

Canonical owner:

  • Iris owns durable continuity and tutor memory persistence.

Primary precursor sources:

  • learner-continuity/*.json
  • tutoring session summaries and timestamps
  • study-workspace session records
  • topic, lesson, course, and evidence references embedded in tutoring context

Required backfill sequence:

  1. Create an Iris-bound memory export format with retention and consent labels.
  2. Backfill continuity snapshots as Iris memory rows with learner, scope, session kind, topic list, evidence summary, and preference state.
  3. Backfill tutoring-session links so memory rows can point back to canonical tutoring_sessions without duplicating the full transcript.
  4. Dual-write new continuity updates to the Iris owner path before read cutover.
  5. Remove Metis-local JSON writes only after Iris parity and deletion/export tests pass.

Special rule:

  • Metis may continue to compute learner-facing continuity summaries, but it must not remain the durable memory store once Iris cutover completes.

6.4 Build-record migration#

Canonical owner:

  • Metis owns CourseBuild, lecture-package, evaluation, export-package, and publication package durability.

Primary precursor sources:

  • lecture-packages/<course-id>/latest.json
  • grounded generation and evaluation manifests
  • export provenance payloads and generated package manifests
  • publication/review queue artifacts that point to build packages

Required backfill sequence:

  1. Create canonical build rows keyed by course, source-bundle version, and build purpose.
  2. Import lecture packages and lesson asset bundles.
  3. Import grounded generation artifacts and evaluation results.
  4. Import export-package metadata and package manifests.
  5. Backfill publication-package rows and review cross-references.
  6. Run parity checks on lesson count, locale variants, asset manifests, provenance summaries, and review readiness state.

Special rule:

  • A build record is not complete unless it can still point back to the source bundle and grounding evidence that produced it.

Canonical owner:

  • Themis owns consent and approval authority.

Primary precursor sources:

  • tutoring teacher-representation policy metadata
  • tutoring representation consent logs
  • review queue artifacts for likeness and voice approvals
  • avatar-release workflow records

Required backfill sequence:

  1. Extract each distinct consent-bearing representation reference from tutoring metadata and review queues.
  2. Backfill a Themis-owned consent record or consent crosswalk for: likeness ID, voice ID, governing artifact, status, expiry, source event, and affected runtime modes.
  3. Link existing tutoring sessions to the canonical consent record IDs.
  4. Keep a Metis compatibility projection for learner disclosure and export, but not a second durable consent truth.

Special rule:

  • Blocked or expired consent states must be preserved exactly during migration; migration must never upgrade a consent decision implicitly.

7. Sequencing and Rollout Order#

The required order is:

  1. LearningSourceBundle
  2. CourseBuild and publication/build records
  3. LearnerMasteryState
  4. TutorSessionMemory
  5. consent records and consent-to-session backfill

Why this order:

  • source bundles are upstream of grounded outlines, generation, and build artifacts
  • build records depend on source and grounding lineage
  • mastery backfill depends on stable assessment and tutoring linkage
  • tutor memory backfill depends on canonical tutoring-session references
  • consent backfill must land last so it can link to final runtime and review artifact IDs rather than precursor placeholders

8. Verification and Rollback#

8.1 Verification requirements#

Each backfill wave must prove:

  • row counts or object counts match the precursor source
  • representative object hashes or semantic equivalence checks pass
  • source-to-target crosswalk rows exist for every migrated object
  • replaying the same migration is idempotent
  • export and deletion paths still honor retention and consent boundaries

8.2 Rollback requirements#

  • No cutover may delete precursor files or embedded metadata until the target owner passes parity checks.
  • The migration ledger must be able to mark a wave as rolled back without destroying the precursor source.
  • Read cutover must be reversible behind a feature flag or owner-backed adapter toggle until two clean verification runs succeed.

9. Exit Criteria For V1-MET-104#

V1-MET-104 is complete only when:

  • this plan remains the canonical Metis migration/backfill reference
  • the owner map aligns with the Metis subsystem matrix
  • every target family has a named precursor store, cutover order, and rollback rule
  • the Metis scorecard continues to point service/persistence reliability work at this migration plan plus the follow-on implementation and observability tasks