Eve library

OSHUN V1 Shared Object Persistence, Migration, and Backfill Plan

Spec

Revision 1

On this page

Status: Active

Date: March 23, 2026

Owners: OSHUN Engineering, OSHUN Platform, OSHUN Product

Related:

  • docs/releases/v1/specs/todos.md
  • docs/releases/v1/reports/subsystem-ownership-matrix.md
  • docs/releases/v1/specs/program-charter.md
  • docs/releases/v1/specs/web-app-admin-v1-prd.md
  • docs/releases/v1/audits/cross-cutting/contract/to-prisma-alignment.md
  • libs/contracts/src/common/contract-prisma-alignment.ts
  • libs/contracts/src/common/contract-prisma-migrations.test.ts
  • apps/oshun/bff/src/nisaba/state.ts
  • apps/isis/output-registry/src/services/output.service.ts
  • apps/lilith/svc-review/src/app.ts
  • apps/lilith/bff/src/utils/translation-review-workflow.ts
  • apps/lilith/svc-spiritual-guidance/src/personas.ts
  • apps/lilith/svc-ai/src/personality-archetypes.ts

1. Purpose#

This document is the authoritative plan for V1-ARCH-078: define persistence, migration, and backfill plans for the new OSHUN V1 Nisaba, notebook, evidence, review, persona, and provenance objects that were added as canonical contracts and owner-bound Prisma models.

This plan is intentionally owner-bound.

When later tasks refer to "shared object storage", they do not authorize a new shell-owned source of truth. Shared objects remain canonically owned by their declared subsystem or domain. The shell and BFF may compose, index, and cache them, but they must not become the durable business-logic owner.

2. Existing Durable Patterns To Reuse#

The repo already contains the migration and backfill patterns this plan should reuse instead of reinventing.

Existing implementation Why it matters
libs/contracts/src/common/contract-prisma-alignment.ts It is the executable manifest of canonical owner boundaries and Prisma targets.
libs/contracts/src/common/contract-prisma-migrations.test.ts It already proves forward and rollback SQL can be generated for the new owner models, even where checked-in migration directories do not yet exist.
apps/isis/output-registry/src/services/output.service.ts It demonstrates idempotent provenance backfill, batch scanning, enrichment, source counting, and marker-based replay safety.
apps/oshun/bff/src/nisaba/state.ts It is the concrete current source for Nisaba passage, notebook, annotation, compare, and grounded-evidence demo state that must be retired from in-memory storage.
apps/lilith/svc-review/src/app.ts It shows the current review queue/item/assignment/escalation model and its file-backed snapshot format.
apps/lilith/bff/src/utils/translation-review-workflow.ts It shows an existing restart-safe but local workflow snapshot model that must be absorbed into the canonical review package owner.
apps/lilith/svc-spiritual-guidance/src/personas.ts and apps/lilith/svc-ai/src/personality-archetypes.ts They are the clearest current persona registries to seed the new canonical persona store.

3. Authoritative Owner Map#

The canonical owner map is already settled by the ownership matrix, PRD, and contract-to-Prisma alignment audit.

Object family Canonical owner Canonical model Current legacy or precursor sources
Nisaba passage data Nisaba libs/nisaba/database/prisma/schema.prisma Passage Nisaba corpus/reference/edition data and the temporary BFF Nisaba reader state
Notebook Sophia libs/sophia/database/prisma/schema.prisma Notebook Nisaba in-memory notebooks, Nisaba PersonalLibraryItem, Nisaba ResearchProject collaboration context
Evidence pack Sophia libs/sophia/database/prisma/schema.prisma EvidencePack Sophia citations/documents, Veritas claim evidence, Nisaba grounded evidence views
Persona Lilith libs/lilith/database/prisma/schema.prisma Persona Lilith persona registries, archetype catalogs, policy-adjacent persona definitions
Provenance bundle Isis libs/isis/database/prisma/schema.prisma ProvenanceBundle Isis GeneratedOutput, Provenance, GenerationJob, LineageEdge, output-registry metadata
Review package Isis libs/isis/database/prisma/schema.prisma ReviewPackage Lilith svc-review, translation review workflow snapshots, admin-review seed queues

Normative rule:

  • Passage, Notebook, EvidencePack, Persona, ProvenanceBundle, and ReviewPackage are the system-of-record rows after cutover.
  • BFF route state, shell caches, compare views, grounded evidence views, and queue summaries are read models or temporary scaffolding, not durable truth.

4. Cross-Cutting Migration Rules#

4.1 ID strategy#

  • If the legacy source already uses the target contract ID shape and that ID is stable, preserve it.
  • If the legacy source uses cuid, slug-like strings, or compound natural keys and the canonical row requires a UUID, generate a deterministic UUIDv5 from: owner + object_type + legacy_source + legacy_id.
  • Every backfill must also persist the original legacy identifier in metadata or a migration ledger so reverse lookup and replay remain possible.

4.2 Replay and idempotency#

  • Every backfill runner must be upsert-based, not insert-only.
  • Every runner must maintain a durable migration ledger containing at least: source name, source cursor or ID, canonical owner, canonical ID, content hash or version hash, status, started at, finished at, and last error.
  • A second run over the same source data must be a no-op unless the source hash changed.
  • Partial-failure recovery must restart from the migration ledger, not from an in-memory cursor.

4.3 Write cutover pattern#

Each object family follows the same cutover stages:

  1. Deploy canonical owner schema and repository path.
  2. Run dry-run extraction and row-shape validation.
  3. Run idempotent backfill into the canonical owner store.
  4. Enable dual-write on new mutations where a legacy write path still exists.
  5. Run parity checks on counts, checksums, and representative object samples.
  6. Switch read paths to the canonical owner.
  7. Remove or quarantine the legacy source-of-truth path.

4.4 What not to backfill#

  • Route caches
  • compare views as standalone durable objects
  • grounded evidence views as standalone durable objects
  • ephemeral pagination/filter state
  • presentation-only queue summaries
  • shell-composed aggregates that can be rebuilt from canonical rows

5. Object-Specific Plans#

5.1 Nisaba Passage And Workspace Data#

Canonical persistence:

  • Passage is the canonical shared passage object and lives in Nisaba.
  • Nisaba-owned adjuncts such as Annotation, AnnotationLayer, ResearchProject, PersonalLibraryItem, and Bookmark remain Nisaba-owned local records unless and until a future shared contract explicitly replaces them.

Migration sources:

  • Nisaba corpus/reference/edition inputs already represented in the Nisaba schema.
  • Temporary shell-facing records in apps/oshun/bff/src/nisaba/state.ts.

Backfill rules:

  • Backfill Passage first. Nothing else in this task block should create a notebook or evidence-pack link to a passage until the passage crosswalk exists.
  • Use normalized reference plus language plus source collection as the primary dedupe key for corpus imports.
  • Preserve passage provenance as first-class data. Corpus or edition lineage must not be flattened away during passage creation.
  • Nisaba Annotation rows remain Nisaba-owned and should be retained locally; they are projected into notebooks rather than migrated out as a new canonical shared object.

Derived-state rule:

  • Nisaba compare views are derived from passages and evidence, not persisted as their own canonical object.
  • Nisaba grounded evidence views are derived from EvidencePack plus passage context, not their own durable store.

5.2 Notebook Backfill Into Sophia#

Canonical persistence:

  • Notebook lives only in Sophia after cutover.

Primary legacy sources:

  • Nisaba BFF notebooks in apps/oshun/bff/src/nisaba/state.ts
  • Nisaba PersonalLibraryItem
  • Nisaba ResearchProject collaborator and workspace context

Migration rules:

  • Create notebook shells first with owner, domain, visibility, and status.
  • Materialize sections and items only after the passage crosswalk is available.
  • Use a deterministic UUIDv5 for notebook IDs derived from the legacy notebook or collection identity.
  • Populate notebook collaborators from explicit project membership when the source is a shared research workspace. Do not invent collaborators for personal-library imports.

Important contract accommodation:

  • The current Nisaba BFF notebook item kinds include annotation, grounded-evidence-view, and compare.
  • The canonical NotebookItemKindSchema does not support those item kinds directly.

Required transforms:

  • legacy passage item -> canonical notebook item with kind: "passage" and a passage link.
  • legacy annotation item -> canonical notebook item with kind: "note", excerpt and note body copied into the item body or summary, and a passage link back to the annotated passage.
  • legacy grounded-evidence-view item -> canonical EvidencePack backfill in Sophia plus an optional notebook note item carrying the user-facing summary and route URL.
  • legacy compare item -> do not create a new durable compare object; store a notebook note summary plus linked passages and regenerate the compare view on demand.

This transform is required because lossless one-to-one item migration is not possible under the current notebook contract.

5.3 Evidence Pack Backfill Into Sophia#

Canonical persistence:

  • EvidencePack lives only in Sophia after cutover.

Primary sources:

  • Sophia Document, DocumentChunk, Citation, Source
  • Veritas ClaimVerification and ClaimEvidence
  • Nisaba grounded evidence view state currently held in apps/oshun/bff/src/nisaba/state.ts

Migration rules:

  • Backfill or resolve supporting Source and Citation records before creating the evidence pack row.
  • Create the pack shell, then hydrate claims, excerpts, citations, retrieval trace, rationale, and review summary.
  • Set notebookId only after the notebook crosswalk exists.
  • Set provenanceBundleId only when the evidence pack directly supports a generated artifact, generated report, or governed synthetic output.

Derived-state rule:

  • Reader-facing "grounded evidence views" in the BFF are projections over the evidence pack plus passage context. They should be regenerated after cutover, not migrated as a second source of truth.

5.4 Persona Backfill Into Lilith#

Canonical persistence:

  • Persona lives only in Lilith after cutover.

Primary sources:

  • apps/lilith/svc-spiritual-guidance/src/personas.ts
  • apps/lilith/svc-ai/src/personality-archetypes.ts
  • related persona-selection or persona-detail catalogs that currently encode runtime persona metadata in code

Migration rules:

  • Seed the canonical persona table from the highest-fidelity persona profile, then enrich it from archetype catalogs where they provide capabilities, style, or teaching metadata absent from the base profile.
  • Use persona slug or source persona key as the deterministic migration key.
  • Leave voiceProfileId and avatarPackId nullable unless a verified Psyche asset registry mapping already exists.
  • Do not mark a persona as approved or published unless there is existing human approval evidence. Default migrated records to review-safe statuses such as draft, review, unsubmitted, or pending_review.
  • Keep code-defined registries as seed fixtures only during migration; after cutover they must become import sources, not runtime truth.

5.5 Provenance Bundle Backfill Into Isis#

Canonical persistence:

  • ProvenanceBundle lives only in Isis after cutover.

Primary sources:

  • Isis GeneratedOutput
  • Isis Provenance
  • Isis GenerationJob
  • Isis LineageEdge
  • output-registry metadata and existing provenance backfill semantics in apps/isis/output-registry/src/services/output.service.ts

Migration rules:

  • Backfill provenance bundles after output, workflow, model, and lineage source data can be resolved.
  • Reuse the output-registry marker pattern so provenance-bundle backfills can be replayed safely and enriched later without duplicating work.
  • If a legacy generated-output identifier is not already a UUID, use the deterministic UUIDv5 crosswalk for primaryArtifactId and keep the original output ID in bundle metadata.
  • Build artifacts, workflow, models, inputs, lineage nodes, lineage edges, watermark signals, disclosure, rights, and integrity from legacy generation data in one canonical bundle rather than splitting them across multiple new tables.

Cutover rule:

  • New generation and output registration flows should dual-write the legacy Provenance record and the canonical ProvenanceBundle until read parity is proven.

5.6 Review Package Backfill Into Isis#

Canonical persistence:

  • ReviewPackage lives only in Isis after cutover.

Primary sources:

  • apps/lilith/svc-review/src/app.ts
  • apps/lilith/bff/src/utils/translation-review-workflow.ts
  • any shell or admin seed queues that currently represent review state outside the canonical owner

Migration rules:

  • Review backfill is last. It depends on canonical artifact IDs from passage, notebook, evidence, persona, and provenance migrations.
  • Preserve stage history, decisions, delegation, SLA timing, and escalation lineage. Review migration is not complete if it only copies current status.
  • Import active and closed reviews separately so reopened or escalated reviews can be replayed safely.
  • For file-backed Lilith review snapshots, import the full snapshot as of the cutover point and then dual-write to the new canonical review package store until the old services are retired.

Artifact-linking rule:

  • artifactRef may preserve the original legacy artifact locator.
  • artifactId must be rewritten to the canonical UUID after the relevant object family crosswalk exists.
  • Dependencies on evidence or provenance must point to canonical EvidencePack and ProvenanceBundle IDs, not legacy local references.

6. Required Sequencing Order#

This is the required migration order for V1-ARCH-079.

Wave 0: Migration Framework#

  • create the durable migration ledger and crosswalk mechanism
  • add dry-run extraction commands and row-shape validation
  • add per-object feature flags for dual-write and read cutover

Wave 1: Foundational Objects#

  • backfill Nisaba Passage
  • backfill Lilith Persona

Rationale:

  • notebooks need passage IDs
  • review packages may need persona artifact IDs

Wave 2: Research Workspace Objects#

  • backfill Sophia Notebook
  • backfill Sophia EvidencePack

Rationale:

  • notebook item transforms depend on passage crosswalks
  • evidence packs may attach to notebooks and passages

Wave 3: Generation Governance Objects#

  • backfill Isis ProvenanceBundle

Rationale:

  • review packages frequently depend on provenance and rights lineage

Wave 4: Approval And Governance Objects#

  • backfill Isis ReviewPackage
  • switch admin review reads to canonical review packages

Rationale:

  • review migration must bind to already-migrated canonical artifact IDs

7. Known Gaps And Required Accommodations#

7.1 Notebook Item Shape Gap#

Current Nisaba notebook items are richer than the canonical notebook item enumeration. annotation, grounded-evidence-view, and compare require transform-based migration rather than one-to-one persistence.

7.2 No Checked-In Prisma Migration Directories Yet#

The repo currently validates migration and rollback SQL by prisma migrate diff and committed schema SQL artifacts rather than by checked-in per-domain Prisma migration folders.

That means V1-ARCH-079 must treat rollout sequencing, SQL generation, backfill, and cutover as one coordinated deployment program. It cannot assume "run the old migration directory" is already available.

7.3 Legacy String IDs Versus Canonical UUID Owners#

Several legacy sources use cuid or ad hoc string IDs while the new canonical objects use UUIDs. Deterministic UUIDv5 crosswalks are mandatory for replay-safe migration and for review/provenance foreign-key integrity.

8. Verification Gates For Implementation#

V1-ARCH-079 and follow-on execution work are not complete until all of the following exist and pass:

  • dry-run extraction for every object family
  • idempotent rerun tests proving no duplicate canonical rows are created
  • parity checks for counts and representative object samples
  • rollback validation for owner-schema changes
  • BFF or adapter integration tests proving reads come from canonical owner stores after cutover
  • partial-failure recovery tests proving migration resumes from the durable ledger
  • audit or provenance verification where review and provenance migrations depend on prior artifact backfills

9. Exit Criteria For V1-ARCH-078#

V1-ARCH-078 is complete only when:

  1. the owner-bound persistence targets are explicit
  2. the legacy migration sources are explicit
  3. the backfill rules are explicit
  4. the sequencing order is explicit
  5. the known contract and rollout gaps are explicit
  6. V1-ARCH-079 can be implemented directly from this document without re-deciding ownership or cutover order