Status: Proposed
Date: 2026-08-27
Authors: V1 Domain Workbenches persistence audit (Phase M, M0.5)
Reviewers: pending — Metis service owner, Platform Identity, Data Governance, Database SRE, Security, and Institutional Integrations
This ADR makes the engineering decision requested by M0.5, but it is not a substitute for owner approval. It deliberately remains Proposed. M0.5.e and the parent M0.5 cell stay open until every named reviewer records approval and the operational acceptance checks in this ADR pass.
Context and audited baseline#
Metis currently connects to the shared oshun_dev PostgreSQL database and lets
Alembic create all 13 tables in the implicit public schema. The same URL is
used by the API pool and by three task-local engines. The production Docker
Compose profile also previously embedded the shared development database URL.
There is no repository layer, no tenant key on any current table, no dedicated
Metis database in local bootstrap, and no row-count or byte-volume snapshot for
a production Metis deployment.
The PostgreSQL development files live on the shared oshun-dev-postgres-data
volume. The global backup runbook includes oshun_dev, but does not name a
dedicated metis database. It specifies continuous managed PITR, nightly
logical dumps, and weekly restore verification; the managed WAL setup and
production execution evidence remain out of band.
The deterministic baseline is
evidence/v1-workbenches/metis-persistence-inventory.json. In particular, an
unknown production row count is an inventory result, not permission to assume
the system is small.
Decision#
Selected strategy: bounded-hybrid.
Provision one dedicated physical PostgreSQL database named metis for
Metis-owned records. Inside it, use owner-scoped schemas and roles: authoring,
learning, assessment, institutional, and integration. Alembic remains
the single DDL authority. No application process may create tables at runtime,
and the public schema must contain no Metis application tables after
migration.
The hybrid boundary is deliberate:
- Metis owns course, curriculum, assessment-authoring, learner-journey, tutoring, assessment-evidence, grade, integrity, and institutional-delivery aggregates that originate in Metis.
- Shared Platform Identity remains canonical for customer identity, tenant membership, entitlements, and sessions. Metis stores opaque references and the minimum authorization projection, never credentials or a second customer profile.
- Content, evidence, memory, model, and runtime objects originating in another OSHUN domain remain canonical in that domain. Metis stores their IDs and, only where a measured read path requires it, a versioned rebuildable projection.
Physical co-location does not expand transaction ownership. A transaction may atomically change records owned by one aggregate boundary. Cross-owner changes use an outbox, idempotent consumers, and reconciliation even when two schemas happen to be in the same database.
Requirements matrix (M0.5.b)#
Every row is normative. “Policy schedule” means the tenant-approved retention schedule recorded by Data Governance; all deletions are blocked by a valid legal hold and audited.
| Aggregate | Canonical owner | Canonical store | Classification | Processing purpose | Policy basis | Access roles | Transaction boundary | Consistency | Performance requirement | Retention and deletion | Data residency | Security requirement |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Course and curriculum authoring | Metis Authoring | metis.authoring |
internal authoring data | create, review, publish, and version learning material | creator contract and product operation | author, reviewer, Metis operator | one course revision and its modules or lessons | serializable publication transition; read projections eventually consistent | authoring commands p95 at most 500 ms excluding media work | active life plus policy schedule; tombstone then purge within 30 days after eligibility | tenant-contracted region | TLS in transit, encrypted storage, tenant row policy, schema-scoped writer |
| Assessment authoring | Metis Authoring | metis.authoring |
confidential assessment material | author and publish instruments, questions, keys, and rubrics | institution contract and product operation | assessment author, reviewer, Metis operator | one assessment revision with question and rubric versions | serializable publication transition; delivery copy version-pinned | authoring commands p95 at most 500 ms | active life plus policy schedule; unpublished drafts purge within 30 days after eligibility | tenant-contracted region | TLS, encrypted storage, tenant row policy, answer-key fields restricted and audited |
| Learner journey | Metis Learning | metis.learning |
protected learner data | enrollment, progress, mastery input, achievement, and continuity | learner request or institution contract | learner self, assigned educator, support with approved purpose | one learner-course enrollment and its progress changes | read-your-writes for learner; authoring metadata eventual by version | progress write p95 at most 250 ms; learner resume read p95 at most 300 ms | policy schedule; erasure within 30 days after eligibility | learner tenant region; no unapproved cross-region replica | TLS, encrypted storage, tenant row policy, subject-scoped access, audited support access |
| Tutoring history | Metis Learning | metis.learning |
protected learner conversation | provide tutoring continuity and safety review | learner request or institution contract | learner self, assigned educator where permitted, safety reviewer | one tutoring session and its ordered messages | ordered per session; summaries and analytics eventual | message persistence p95 at most 200 ms | raw transcript schedule no longer than tenant policy; derived summaries independently expirable | learner tenant region | TLS, encrypted storage, tenant row policy, content access audited, prompt projections minimized |
| Assessment attempt evidence | Metis Assessment Evidence | metis.assessment |
restricted assessment evidence | record immutable attempt inputs and grading provenance | institution contract and academic-record policy | learner self where policy permits, grader, appeal reviewer | one attempt with submitted answers and evidence manifest | append-only attempt acceptance; grade and integrity decisions eventual | submission acceptance p95 at most 300 ms before asynchronous grading | academic-record policy schedule; legal hold capable; purge within 30 days after eligibility | institution-contracted region | TLS, field encryption for answers, tenant row policy, append-only audit, no authoring access |
| Grade record | Metis Grade Authority | metis.assessment |
restricted education record | issue, revise, release, and appeal grades | institution contract and education-record policy | learner self after release, grader, registrar, appeal reviewer | one versioned grade decision | monotonic versions; release requires attempt reference; downstream views eventual | released-grade read p95 at most 300 ms | academic-record policy schedule; corrections retained as superseded versions; legal hold capable | institution-contracted region | TLS, encrypted storage, tenant row policy, dual-control override, every read and mutation audited |
| Integrity record | Metis Integrity Authority | metis.assessment |
highly restricted integrity record | investigate and decide assessment-integrity signals | institution contract and documented integrity policy | integrity reviewer and appeal reviewer only | one versioned integrity case and its evidence references | case decision independent from grade transaction; effects published by event | reviewer case read p95 at most 500 ms | integrity-policy schedule; legal hold capable; allegation data deleted when no longer justified | institution-contracted region; no analytics export by default | TLS, field encryption, dedicated role, tenant row policy, reason-bound audited access |
| Institutional configuration | Metis Institutional Integrations | metis.institutional |
confidential tenant configuration | LMS mappings, delivery policy, issuer configuration, and roster synchronization | institution contract | institution admin and integration operator | one tenant integration configuration version | strong for configuration version; delivery caches eventual | configuration reads p95 at most 200 ms | contract life plus 90 days unless policy requires earlier deletion | institution-contracted region | TLS, secret references not secret values, tenant row policy, privileged mutations audited |
| Learning activity statement | Metis Learning | metis.learning |
protected learner activity | record xAPI or cmi5 activity and support authorized export | learner request or institution contract | learner self where required, assigned educator, integration operator | one idempotent statement identified by statement ID | append-only and idempotent; export cursor monotonic | ingest p95 at most 200 ms | tenant learning-record schedule; correction by voiding event; legal hold capable | learner tenant region | TLS, encrypted payload, tenant row policy, actor and result fields excluded from general logs |
| Identity and tenant reference | Shared Platform Identity | external identity database plus metis.integration reference |
restricted identity and authorization projection | authorize a Metis request and route it to one tenant | account contract and security obligation | identity service, Metis authorization adapter | identity owner transaction only; Metis stores received version separately | event-driven projection with fail-closed authorization on stale or missing policy | authorization projection lookup p95 at most 50 ms | projection expires or deletes within 24 hours of owner revocation event | same or stricter residency than source tenant | signed events, opaque IDs, no password or refresh-token copy, least-privilege reader |
Ownership consequences for the current tables#
| Current table or field family | Target disposition |
|---|---|
courses, modules, lessons, assessments, questions |
Move to versioned aggregates in authoring; assessment keys receive restricted-column access. |
enrollments, progress, achievements, tutoring_sessions, tutoring_messages, xapi_statements |
Move to tenant-keyed aggregates in learning; xAPI is learner activity, not institutional configuration. |
submissions |
Split into assessment_attempts, grade_records, and integrity_records; preserve one lineage ID across the split. |
users credentials and profile columns |
Retire after Platform Identity cutover; replace with a minimal versioned identity and tenant authorization reference in integration. |
| Cross-domain content or evidence | Keep the originating canonical ID; allow only a rebuildable projection with source version and payload hash. |
Option analysis (M0.5.c)#
| Option | Isolation | Operational cost | Migrations | Joins and events | Backup and restore | Scaling | Ownership clarity | Decision |
|---|---|---|---|---|---|---|---|---|
Shared oshun_dev public schema |
poor: credentials, schema, locks, and restore blast radius are shared | lowest initially, highest contention and incident coordination | simple until unrelated migrations collide | easy joins encourage hidden coupling; events often skipped | cannot restore Metis independently without shared-database surgery | vertical contention with every tenant and domain | poor: physical access implies false ownership | rejected |
| Dedicated all-in-one Metis database | strong physical isolation | moderate database and on-call cost | one clean Alembic chain, but imports adjacent canonical copies | local joins are easy; cross-domain changes tempt shadow copies | independent database PITR and logical restore | independent compute and storage, but all record classes scale together | medium: Metis appears to own identity and originating-domain facts | rejected |
| Bounded hybrid with dedicated Metis database | strong physical isolation plus logical owner boundaries | moderate database, event relay, and reconciliation cost | staged move to owner schemas with explicit compatibility period | joins stay inside one owner; cross-owner flow is event or versioned projection | Metis database restores independently; external projections rebuild from owners | schemas and workloads can split later without ID changes | strong: one owner per aggregate and no shadow canonical copy | selected |
The bounded hybrid pays a real eventing and reconciliation cost. That cost is preferred to the irrecoverable ambiguity of shared ownership or a locally convenient copy that becomes a second system of record.
Integration design (M0.5.d)#
| Topic | Decision |
|---|---|
| Canonical IDs | Preserve current immutable UUIDv4 IDs for Metis-owned records. Accept canonical opaque string IDs from external owners. Never mint a replacement for an external ID. Add tenant_id from Platform Identity to every tenant-scoped aggregate before cutover. |
| Cross-database references | Store the owner-qualified ID and expected owner version. PostgreSQL foreign keys stop at the database boundary; adapters validate existence and authorization. A missing owner is an unavailable dependency, not evidence that Metis owns the record. |
| Event envelope | Every event carries event_id, event_type, schema_version, aggregate_id, aggregate_version, tenant_id, occurred_at, correlation_id, causation_id, and producer. Payloads are minimized and classified. |
| Transaction and delivery | Write the aggregate and an outbox row in one owner transaction. Relay at least once. Consumers use an inbox keyed by event_id, reject version regression, and reconcile gaps. No distributed transaction or simultaneous dual-authoritative write is allowed. |
| Projections | A consumer projection records source_owner, source_id, source_version, source_event_id, payload_hash, and projected_at. It is rebuildable and cannot accept local canonical mutations. |
| Corrections, deletion, and holds | Owners publish versioned correction, tombstone, erasure-eligible, and legal-hold state events. Consumers update or suppress projections idempotently and retain audit proof without retaining deleted payload. |
| Shadow-copy prohibition | Credentials, full external profiles, raw protected learner history, source documents, and external canonical payloads may not be copied into Metis authoring records. A cache without source identity, version, hash, expiry, and rebuild procedure is a prohibited shadow copy. |
| Submission split | One migration lineage ID links attempt evidence, grade versions, and integrity cases. Their owners transact separately and exchange decisions through events; a grade rollback cannot erase attempt evidence or integrity history. |
Migration and rollback#
| Phase | Forward action | Verification gate | Rollback |
|---|---|---|---|
| 0. Contain | Require an explicit production database URL and reject the known shared oshun_dev database in production settings. |
Configuration unit tests and production Compose interpolation check pass. | Remove the guard only through a reviewed emergency change; this phase moves no data. |
| 1. Provision | Create the metis database, owner schemas, least-privilege roles, migrations, outbox, inbox, backup registration, and residency placement. |
Empty-database migration, role-denial, PITR configuration, and synthetic restore tests pass. | Drop only the new empty database after preserving provisioning logs. |
| 2. Expand and capture | Add tenant keys, aggregate versions, lineage IDs, and source-side outbox capture to the legacy tables. The shared database remains authoritative. | Existing traffic remains green; every accepted write has a same-transaction outbox row. | Stop relay and remove expanded nullable columns in a reviewed backward migration. |
| 3. Backfill | Snapshot by stable key, copy into target owner schemas, then relay changes from the captured cursor. Split each submission without changing its lineage ID. | Per-tenant row counts, canonical payload hashes, relationship counts, and sampled semantic comparisons match; no orphan or version regression exists. | Discard target data and restart from a recorded source cursor; the legacy owner is unchanged. |
| 4. Cut over | Quiesce writes, drain source-to-target relay, verify hashes, switch all API and worker connection secrets together, then make target authoritative. Start a target-to-legacy compatibility sink for the rollback window. | API, worker, migration-head, authorization, latency, error-rate, outbox-lag, and restore-smoke gates pass. | Quiesce, drain target-to-legacy events, verify parity, and point all consumers back to legacy. Never write authoritatively to both sides. |
| 5. Contract | After the approved rollback window and restore drill, revoke Metis access to shared oshun_dev, archive the compatibility sink, and drop legacy tables in a separate migration. |
Named owners sign off; no production consumer or backup restore references the legacy tables. | Restore the pre-contract backup into an isolated recovery database and replay owner events; do not silently recreate shadow tables. |
Migration controls must be tenant-aware and resumable. A single failed tenant halts that tenant's cutover without requiring a global rollback. The production row and byte baseline must be captured before capacity approval for phase 1.
Operations and acceptance ownership#
| Responsibility | Accountable owner | Required evidence before acceptance |
|---|---|---|
| Aggregate and API behavior | Metis service owner | contract tests, transaction tests, outbox and inbox idempotency tests |
| Identity and tenant contract | Platform Identity | signed ID and membership event contract, revocation-lag test, stale-policy fail-closed test |
| Classification, retention, deletion, residency, legal hold | Data Governance | approved schedule per data class, residency placement record, erasure and hold drill |
| Database provisioning, backup, restore, capacity, and failover | Database SRE | IaC review, encrypted secret provisioning, PITR evidence, logical backup, synthetic restore, capacity baseline |
| Roles, encryption, audit, and incident response | Security | role-denial matrix, encryption evidence, audit access review, threat model |
| LMS and xAPI delivery contracts | Institutional Integrations | tenant-isolation, replay, ordering, export, and voiding-statement tests |
Operational acceptance requires all rows above. A passing application test suite alone cannot ratify a persistence boundary.
Rejected alternatives#
- Keeping
oshun_devbecause it is already deployed was rejected: deployment convenience does not provide tenancy, ownership, independent recovery, or a production capacity baseline. - A database per table or aggregate was rejected for V1: it maximizes secrets, pools, migrations, and restore coordination before workload measurements justify the split.
- A single Metis database containing canonical copies of Platform Identity or other domains was rejected: local joins do not justify a second owner.
- Synchronous dual writes and two-phase commit were rejected: they enlarge the failure domain and still leave recovery semantics ambiguous.
- Permanent foreign-data wrappers or cross-database SQL joins were rejected as a domain contract: they couple availability and bypass versioned APIs/events.
- A flag-day migration was rejected because the current production data volume is unknown and rollback would be unverifiable.
Ratification record#
| Reviewer | Status | Evidence |
|---|---|---|
| Metis service owner | pending | implementation and contract review required |
| Platform Identity | pending | identity, tenant, and revocation contract review required |
| Data Governance | pending | classification, retention, residency, deletion, and hold approval required |
| Database SRE | pending | topology, capacity, backup, restore, and rollback approval required |
| Security | pending | threat model, roles, encryption, and audit approval required |
| Institutional Integrations | pending | LMS and xAPI contract review required |
The status may change to Accepted only when every ratification row is approved with linked evidence. Until then, M0.5.e remains incomplete.