# OSHUN V1 Metis Persistence, Migration, and Backfill Plan

Status: Active

Date: April 10, 2026

Owners: OSHUN Engineering, OSHUN Platform, OSHUN Safety

Related:

- `docs/releases/v1/specs/todos.md`
- `docs/releases/v1/reports/metis-v1-scorecard.md`
- `docs/releases/v1/reports/metis-v1-workstream-tracking.md`
- `docs/releases/v1/reports/metis-subsystem-ownership-matrix.md`
- `docs/releases/v1/specs/shared-object-persistence-migration-backfill-plan.md`
- `docs/releases/v1/specs/metis-learning-source-bundle-contract.md`
- `docs/releases/v1/specs/metis-grounding-pack-contract.md`
- `docs/releases/v1/specs/metis-course-build-contract.md`
- `docs/releases/v1/specs/metis-publication-package-contract.md`
- `docs/releases/v1/specs/metis-voice-likeness-consent-record-contract.md`
- `services/metis/src/metis/services/source_ingestion_service.py`
- `services/metis/src/metis/services/learner_mastery_service.py`
- `services/metis/src/metis/services/learner_continuity_service.py`
- `services/metis/src/metis/services/lecture_generation_service.py`
- `services/metis/src/metis/services/tutoring_service.py`
- `services/metis/src/metis/tasks/export.py`

## 1. Purpose

This document is the authoritative plan for `V1-MET-104`: define persistence,
migration, and backfill plans for the new Metis source-bundle, mastery,
tutor-memory, build-record, and consent-record object families.

It exists because Metis now has canonical contracts and live service logic for
those families, but the current repo still spans three different durability
patterns:

- SQLAlchemy-backed relational rows for courses, assessments, enrollments,
  submissions, tutoring sessions, and tutoring messages
- file-backed JSON manifests rooted under `METIS_SOURCE_INGESTION_STORAGE_DIR`
- metadata blobs embedded inside existing session, message, or export payloads

This plan defines how those precursor stores are retired without allowing
ownership drift away from the subsystem matrix.

## 2. Existing Durable Patterns To Reuse

Metis should reuse the repo's existing migration and backfill patterns rather
than invent a Metis-only format.

| Existing implementation                                                       | Why it matters                                                                                                                       |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `docs/releases/v1/specs/shared-object-persistence-migration-backfill-plan.md` | It is the canonical Oshun-wide migration and cutover template.                                                                       |
| `libs/contracts/src/common/contract-prisma-migrations.test.ts`                | It establishes the expected migration proof standard for owner-bound contract persistence.                                           |
| `apps/isis/output-registry/src/services/output.service.ts`                    | It demonstrates idempotent backfill, marker-based replay, and provenance-safe enrichment.                                            |
| `services/metis/src/metis/services/source_ingestion_service.py`               | It is the current raw source-package durability layer and the highest-fidelity precursor for `LearningSourceBundle` migration.       |
| `services/metis/src/metis/services/learner_mastery_service.py`                | It is the current mastery snapshot writer and the precursor source for canonical learner mastery persistence.                        |
| `services/metis/src/metis/services/learner_continuity_service.py`             | It is the current continuity-memory writer and the precursor source for Iris-owned tutor/session memory migration.                   |
| `services/metis/src/metis/services/lecture_generation_service.py`             | It is the current durable lecture/build package writer and the precursor source for Metis build and publication records.             |
| `services/metis/src/metis/services/tutoring_service.py`                       | It contains the current consent-bearing tutoring policy, handoff, runtime-fallback, and representation metadata that must be split.  |
| `services/metis/src/metis/tasks/export.py`                                    | It shows the current export-grade package assembly and the build/export provenance surface that backfill verification must preserve. |

## 3. Authoritative Owner Map

The owner map is already settled by
`docs/releases/v1/reports/metis-subsystem-ownership-matrix.md`.

| Object family                                            | Canonical owner | Canonical durable target                                                                 | Current precursor sources                                                                                       |
| -------------------------------------------------------- | --------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `LearningSourceBundle`                                   | Sophia          | Sophia-owned source-bundle rows plus migration ledger and source-to-bundle crosswalk     | `source_ingestion_service.py`, extraction/refresh/outline/generation/concept graph manifests under file storage |
| `LearnerMasteryState`                                    | Metis           | Metis-owned mastery rows and event ledger keyed to learner, topic, course, and session   | `learner_mastery_service.py`, assessment submissions, tutoring turns, analytics-derived mastery context         |
| `TutorSessionMemory`                                     | Iris            | Iris-owned durable session-memory and continuity rows with consent-aware retention scope | `learner_continuity_service.py`, tutoring-session summaries, session/message metadata                           |
| `CourseBuild` and build records                          | Metis           | Metis-owned build, lecture-package, publication, evaluation, and export package rows     | `lecture_generation_service.py`, source generation/evaluation outputs, export task payloads                     |
| `VoiceLikenessConsentRecord` and related consent records | Themis          | Themis-owned consent and approval records with Metis crosswalk references                | tutoring-session teacher-representation metadata, consent logs, admin review queue artifacts                    |

Normative rules:

- Metis may continue to orchestrate these objects, but it must not become the
  durable owner for Sophia, Iris, or Themis records.
- File-backed manifests under `METIS_SOURCE_INGESTION_STORAGE_DIR` are a
  precursor store, not an acceptable long-term system of record.
- Embedded metadata JSON in tutoring messages is a compatibility seam, not the
  final durable model for memory or consent state.

## 4. Current Precursor Storage Surfaces

### 4.1 File-backed precursor stores

The current file-backed state under `METIS_SOURCE_INGESTION_STORAGE_DIR`
contains the highest-fidelity precursor data for several new object families:

- source package manifests and artifacts
- extraction results
- review state and usage-policy state
- grounded outlines
- grounded generation artifacts
- retrieval and concept-graph artifacts
- study-workspace projections
- learner mastery snapshots in `learner-mastery/*.json`
- learner continuity memory in `learner-continuity/*.json`
- lecture packages in `lecture-packages/<course-id>/latest.json`

### 4.2 Relational precursor stores

Metis already has relational rows for:

- `courses`, `modules`, `lessons`
- `assessments`, `questions`, `submissions`
- `enrollments`, `progress`, `achievements`
- `tutoring_sessions`, `tutoring_messages`
- `users.preferences`

These rows already carry source IDs, topic labels, progress markers, lesson
links, grading outputs, and tutoring metadata needed to backfill the new object
families.

### 4.3 Embedded metadata seams to retire

The following data is currently embedded and must be lifted into explicit
records or crosswalks:

- tutoring delivery-fallback policy
- oversight and integrity policy
- teacher representation and live runtime metadata
- consent logs and consent summaries
- export package provenance summaries when only embedded in export payloads

## 5. Cross-Cutting Migration Rules

### 5.1 ID and crosswalk strategy

- Preserve stable IDs where the current contract already exposes them and the
  owning subsystem accepts them.
- Otherwise generate deterministic UUIDv5 values from:
  `metis + object_family + precursor_store + precursor_id`.
- Persist a migration crosswalk for every migrated row: precursor store,
  precursor ID, canonical owner, canonical ID, content hash, migrated at, and
  latest verification status.

### 5.2 Replay and idempotency

- Every backfill runner must be upsert-based.
- Every runner must maintain a migration ledger with: object family, source
  cursor, canonical owner, canonical ID, source hash, started at, finished at,
  status, and last error.
- Re-running the same migration with the same source hash must be a no-op.
- A changed source hash must create either an in-place update or a new versioned
  canonical row according to the target contract's versioning rules.

### 5.3 Cutover stages

Each Metis object family follows the same stages:

1. Define target schema and migration ledger.
2. Run dry-run extraction and contract-shape validation.
3. Backfill from file-backed or embedded precursor sources.
4. Enable dual-write where the old write path is still active.
5. Run parity checks on counts, hashes, and representative samples.
6. Switch reads to the canonical owner or owner-backed adapter.
7. Freeze and then retire the legacy precursor path.

### 5.4 What must not be backfilled

- transient tutoring websocket/runtime tokens
- shell-only continuation cards
- in-browser derived analytics summaries
- learner notification projections
- grounded inspector view state
- admin dashboard summary tiles

## 6. Object-Specific Plans

### 6.1 `LearningSourceBundle` migration

Canonical owner:

- Sophia owns the durable bundle and review lineage.

Primary precursor sources:

- `SourceIngestionService` package manifests
- extraction manifests
- review and usage-policy state
- refresh/version-diff manifests
- connector previews, concept graphs, outlines, and generation manifests

Required backfill sequence:

1. Register a source-bundle migration ledger and scope crosswalk.
2. Import raw package shell rows first: scope, requestor, source version, item
   inventory, package status.
3. Attach extraction and citation manifests.
4. Attach review history and usage-policy state.
5. Attach derived asset references: outline, generation artifacts, retrieval
   indices, concept graphs, refresh/version history, study-workspace references.
6. Mark bundle rows as cutover-ready only after all derived artifact pointers
   are reproducible from the canonical row.

Special rule:

- Metis must not keep a parallel long-term copy of source trust and approval
  state after Sophia cutover. Metis may cache bundle summaries for learner and
  admin views, but those caches are rebuildable read models only.

### 6.2 `LearnerMasteryState` migration

Canonical owner:

- Metis owns mastery semantics and mastery-state decisions.

Primary precursor sources:

- `learner_mastery/*.json`
- assessment submissions and fairness/calibration outputs
- tutoring turn updates recorded through `LearnerMasteryService`
- adaptive-learning and analytics-derived mastery recommendations

Required backfill sequence:

1. Create a mastery ledger keyed by learner, focus topic, course, and source
   evidence version.
2. Import the current mastery snapshot for each learner.
3. Rebuild event history where available from: tutoring turns, assessment
   attempts, retry outcomes, and misconception links.
4. Attach prerequisite and misconception graph references used in adaptive
   sequencing.
5. Recompute and compare derived mastery context before cutover.

Parity checks:

- same weakest-skill set
- same blocked-prerequisite set
- same mastery probability bucket for representative topics
- same adaptive next-step recommendation class

### 6.3 `TutorSessionMemory` migration

Canonical owner:

- Iris owns durable continuity and tutor memory persistence.

Primary precursor sources:

- `learner-continuity/*.json`
- tutoring session summaries and timestamps
- study-workspace session records
- topic, lesson, course, and evidence references embedded in tutoring context

Required backfill sequence:

1. Create an Iris-bound memory export format with retention and consent labels.
2. Backfill continuity snapshots as Iris memory rows with learner, scope,
   session kind, topic list, evidence summary, and preference state.
3. Backfill tutoring-session links so memory rows can point back to canonical
   `tutoring_sessions` without duplicating the full transcript.
4. Dual-write new continuity updates to the Iris owner path before read cutover.
5. Remove Metis-local JSON writes only after Iris parity and deletion/export
   tests pass.

Special rule:

- Metis may continue to compute learner-facing continuity summaries, but it must
  not remain the durable memory store once Iris cutover completes.

### 6.4 Build-record migration

Canonical owner:

- Metis owns `CourseBuild`, lecture-package, evaluation, export-package, and
  publication package durability.

Primary precursor sources:

- `lecture-packages/<course-id>/latest.json`
- grounded generation and evaluation manifests
- export provenance payloads and generated package manifests
- publication/review queue artifacts that point to build packages

Required backfill sequence:

1. Create canonical build rows keyed by course, source-bundle version, and build
   purpose.
2. Import lecture packages and lesson asset bundles.
3. Import grounded generation artifacts and evaluation results.
4. Import export-package metadata and package manifests.
5. Backfill publication-package rows and review cross-references.
6. Run parity checks on lesson count, locale variants, asset manifests,
   provenance summaries, and review readiness state.

Special rule:

- A build record is not complete unless it can still point back to the source
  bundle and grounding evidence that produced it.

### 6.5 Consent-record migration

Canonical owner:

- Themis owns consent and approval authority.

Primary precursor sources:

- tutoring teacher-representation policy metadata
- tutoring representation consent logs
- review queue artifacts for likeness and voice approvals
- avatar-release workflow records

Required backfill sequence:

1. Extract each distinct consent-bearing representation reference from tutoring
   metadata and review queues.
2. Backfill a Themis-owned consent record or consent crosswalk for: likeness ID,
   voice ID, governing artifact, status, expiry, source event, and affected
   runtime modes.
3. Link existing tutoring sessions to the canonical consent record IDs.
4. Keep a Metis compatibility projection for learner disclosure and export, but
   not a second durable consent truth.

Special rule:

- Blocked or expired consent states must be preserved exactly during migration;
  migration must never upgrade a consent decision implicitly.

## 7. Sequencing and Rollout Order

The required order is:

1. `LearningSourceBundle`
2. `CourseBuild` and publication/build records
3. `LearnerMasteryState`
4. `TutorSessionMemory`
5. consent records and consent-to-session backfill

Why this order:

- source bundles are upstream of grounded outlines, generation, and build
  artifacts
- build records depend on source and grounding lineage
- mastery backfill depends on stable assessment and tutoring linkage
- tutor memory backfill depends on canonical tutoring-session references
- consent backfill must land last so it can link to final runtime and review
  artifact IDs rather than precursor placeholders

## 8. Verification and Rollback

### 8.1 Verification requirements

Each backfill wave must prove:

- row counts or object counts match the precursor source
- representative object hashes or semantic equivalence checks pass
- source-to-target crosswalk rows exist for every migrated object
- replaying the same migration is idempotent
- export and deletion paths still honor retention and consent boundaries

### 8.2 Rollback requirements

- No cutover may delete precursor files or embedded metadata until the target
  owner passes parity checks.
- The migration ledger must be able to mark a wave as rolled back without
  destroying the precursor source.
- Read cutover must be reversible behind a feature flag or owner-backed adapter
  toggle until two clean verification runs succeed.

## 9. Exit Criteria For `V1-MET-104`

`V1-MET-104` is complete only when:

- this plan remains the canonical Metis migration/backfill reference
- the owner map aligns with the Metis subsystem matrix
- every target family has a named precursor store, cutover order, and rollback
  rule
- the Metis scorecard continues to point service/persistence reliability work at
  this migration plan plus the follow-on implementation and observability tasks
