Data Architecture and Tenancy

How V1 stores data, isolates tenants, and routes every request to the right data plane. This page is for engineers and operators who need to reason about where a piece of data lives, who is allowed to touch it, and how the foundation libraries enforce those boundaries. It sits in the Foundations layer of V1 — below the Customer-Facing Domains and the substrates (Sophia, Iris, Psyche, Lilith, Isis, Aje), and is the substrate the BFF request lifecycle runs on top of.

Everything below is backed by real code in the monorepo. Where a capability is spec-only, contract-level, or provider-gated, that is called out explicitly — honest "planned/gated" beats a fake "shipped."

Where the data lives: per-domain databases#

V1 deliberately avoids a single shared schema. Each domain owns its own PostgreSQL database so that ownership, migrations, residency, and deletion are bounded surfaces rather than a tangled monolith.

Local development#

Local development uses domain-isolated PostgreSQL databases on a single cluster (per CLAUDE.md "Local Development"):

These are the seven databases enumerated in the dev profile. The docker-compose.dev.yml core stack provisions PostgreSQL (with pgvector), Redis, MinIO, and Mailpit; the same connection strings are read from the root .env / .env.example.

Production#

Production extends the dev layout with domain-specific databases for the six customer-facing domains and the Metis service databases. Connection URLs follow a <DOMAIN>_DATABASE_URL convention documented in .env.example. Confirmed examples in the file today:

TARA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/tara
NISABA_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/nisaba
CYBELE_DATABASE_URL=postgresql://oshun:oshun_dev@cybele-pgbouncer:5432/cybele
MAAT_DATABASE_URL=postgresql://oshun:oshun_dev@pgbouncer:6432/maat
LAKSHMI_DATABASE_URL=postgresql://oshun:oshun_dev@lakshmi-pgbouncer:6438/lakshmi

The pattern generalizes to the rest of the domains (ARETE_*, NYX_*, VERITAS_*, METIS_*); the production envs supply those URLs even though the checked-in .env.example only spells out a representative subset. The cybele-pgbouncer / lakshmi-pgbouncer host names show the production shape: every database sits behind its own PgBouncer.

Why one database per domain. A domain that owns its store can run its own migrations, enforce its own tombstone policy, and answer a deletion or DSAR request without coordinating a cross-domain transaction. Cross-domain effects are carried as events (see Communication Patterns), not as foreign keys into another domain's tables — so the blast radius of any one schema change stays inside that domain.

Shared cluster features#

• pgvector for embedding storage — used by Sophia retrieval, the Nisaba concept graph, and recommendation surfaces. It rides on the same PostgreSQL cluster, so embeddings live next to their owning domain's rows.
• TimescaleDB for time-series telemetry where required. The LAKSHMI_* envs encode this pattern: LAKSHMI_DIRECT_DATABASE_URL points at a lakshmi-timescale host, with split ingest / analytics dev URLs (LAKSHMI_DEV_INGEST_DATABASE_URL, LAKSHMI_DEV_ANALYTICS_DATABASE_URL).
• PgBouncer in front of every production database for connection pooling (e.g., tara-pgbouncer, cybele-pgbouncer, lakshmi-pgbouncer). The dev pooler listens on 6432; production poolers use per-domain ports (lakshmi-pgbouncer:6438).

Optional stores (per Docker dev profiles)#

These are opt-in via docker compose --profile <name>:

• Elasticsearch (search profile) — universal search backbone. See Search, Discovery, and Knowledge Graph.
• Qdrant (vectors profile) — vector store for high-cardinality similarity (Sophia retrieval, generation lineage) where pgvector's locality is not the right tradeoff.
• Neo4j (graph profile) — graph store for concept-graph reads (Nisaba) and Maat-style consequence chains.
• Kafka (streaming profile) — streaming for high-volume cross-domain events when the default event bus's capacity is exceeded. (Note: the default event bus is not Kafka and not native Redis Streams — see The event bus is not Redis Streams below.)
• MinIO in dev, S3 in production — object storage for generated media, Living-Scenes renders, source ingestion, and exports.

The persistence foundation — libs/oshun/persistence/#

Every domain database is generated from contracts, not hand-authored. The persistence library is the gate that keeps the Zod contracts and the physical schema in lockstep.

• prisma-renderer.ts renders a Prisma schema from the contract registry, and zod-prisma-introspection.ts walks Zod schemas to extract the field shape the renderer needs. Schemas drive the database, not the reverse.
• contract-persistence-registry.ts is the registry of which contract maps to which persistent table. Its companion contract-persistence-registry.test.ts is a drift test: it fails the build the moment a Zod field and its persistence column diverge.
• tombstone-semantics.ts and tombstone-semantics.test.ts enforce that deletion on every user-data table is a soft tombstone that propagates and is audit-logged. There is no silent re-creation — a tombstoned row stays tombstoned.
• index-requirements.ts and index-requirements.test.ts validate per-table required indexes via EXPLAIN checks in CI, so a query that should hit an index can't silently regress to a sequential scan.
• migration-plan.ts and migration-plan.test.ts keep migration plans idempotent and dry-runnable.

Beyond that quartet, the same library hosts the DSAR / erasure runtime that turns a privacy request into a real cascade:

• dsar-deletion-cascade.ts (with unit and integration tests) computes the deletion fan-out across owned tables.
• dsar-erasure-runtime.ts (with unit and integration tests) executes the erasure against the live store.

This is why deletion in V1 is a first-class, tested operation rather than a DELETE FROM afterthought — see Trust, Safety, and Privacy for the customer-facing privacy surface.

Residency: every request carries a data plane#

Residency is enforced, not advisory. The enforcement lives in libs/shared/data-residency/, whose src/ contains these modules (the audit bridge below is an injected seam exercised by audit-platform-bridge.spec.ts, not a standalone module file):

The contract is simple and strict:

• Every request carries a residency context derived from the user's primary data plane (their tenant's homeZone). The BFF, generation pipelines, and provider calls all honor that context.
• Cross-region access is explicit consent only. A cross-region read or write is recorded as a ConsentRecord (owned by Iris) before it is allowed. Absent the consent record, the enforcer requires consent rather than silently failing open.

This pairs with the BFF middleware order: tenant resolves first (so the home zone is known), then the residency enforcer can short-circuit the rest of the pipeline on a policy decision before any body validation or adapter call happens. See the request-lifecycle diagram in Communication Patterns.

Tenancy: the graph that anchors authorization, residency, and consent#

The tenancy graph is the spine of V1's authorization model. Tenants form an inheritance hierarchy with explicit policy-override records; users hang off a tenant and carry their own consent records and memory scopes. Bounding everything to this graph is what makes DSAR and admin inspection finite, auditable surfaces rather than open-ended sweeps.

Tenancy contracts#

• Tenant, TenantHierarchyEdge, and TenantPolicyInheritance contracts (§1.2) describe the tenancy graph used by the Tenant Console.
• Tenant feature flags and experiments are scoped — they never bleed across tenant boundaries.
• The tenant-isolation test suite at tests/security/tenant-isolation/ is launch-gating: a cross-tenant leak fails the release.

The tenant / consent / memory-scope model#

Reading the graph:

• Tenant.homeZone is the field the residency enforcer reads to decide the request's data plane. A child tenant inherits its parent's policies unless a TenantPolicyInheritance row marks override = true with source = 'self'.
• TenantHierarchyEdge.relation distinguishes an org from a cohort from a lab, so a university (org) can own cohorts that own per-lab tenants without any of them sharing flags or cache keys.
• ConsentRecord is the audit anchor for memory and cross-region access. A revocation sets revokedAt, and iris.consent.changed fans out to every domain to re-evaluate memory scopes (see the event table in Communication Patterns).
• MemoryScope.kind bounds what assistant/Iris memory a surface may read: profile, session, notebook, operator-copilot, or tenant. Admin inspection and DSAR only ever traverse a user's own scopes.

The role and scope model — @oshun/platform-foundations#

Tenancy answers which data plane; the role model answers which operations. Both live under one package: libs/oshun/platform-foundations/, whose src/index.ts re-exports exactly nine subsystems — the single home for these foundations that the rest of the docs reference piecemeal:

export * from './service-discovery/index';
export * from './public-api/index';
export * from './shared-contracts/index';
export * from './role-model/index';
export * from './step-up/index';
export * from './secrets/index';
export * from './configs/index';
export * from './rollback/index';
export * from './abuse-controls/index';

Canonical roles (§28)#

libs/oshun/platform-foundations/src/role-model/role-model.ts defines CANONICAL_ROLES — ten roles, more than the prose elsewhere implies. The authoritative list (use these exact slugs, not looser names like "support" or "admin leadership"):

Two roles that older prose tends to flatten away are real, distinct canonical roles here: creator (a customer who can reach creator.studio) and tenant-admin (scoped to the tenant console, not global leadership).

The module also exports SCOPE_KEYS (the 21-entry scope vocabulary above), scopesFor(role) which returns a ReadonlySet<ScopeKey>, and filterFieldsByRole() for least-privilege field projection — so a support agent and a privacy operator can read the same record yet see different fields. This is the function-level authorization the role model exists to enforce.

Public API: OAuth 2.1 / PKCE (§27)#

libs/oshun/platform-foundations/src/public-api/oauth.ts is the OAuth 2.1 / PKCE surface for first-party and partner clients. It is contracts- and state-machine-level — token types, PKCE validation patterns, and refresh/revocation surface are implemented and tested; this module is not itself a deployed authorization server. Confirmed shape:

• TOKEN_TYPES = ['access', 'refresh'].
• The PKCE code-challenge regex PKCE_CODE_CHALLENGE_PATTERN = /^[A-Za-z0-9_-]{43,128}$/ enforces a 43–128-char base64url challenge.
• Error codes include 'pkce-required' and 'pkce-verification-failed', so a missing or mismatched verifier fails loud rather than degrading to an unsafe grant.

The other foundation seams#

The remaining seven exports round out the platform: service-discovery, shared-contracts, step-up (step-up auth for sensitive operations), secrets, configs, rollback, and abuse-controls.

Auth primitives — libs/shared/auth-primitives/#

Below the platform package sits a richer set of primitives than "JWT and session" suggests. The src/ exports include:

• jwt.ts, api-key.ts, password.ts — credential primitives.
• oauth-client.ts, oauth-revoke.ts, token-refresh.ts, token-audit.ts — the OAuth client side, revocation, refresh, and an audit trail for token events.
• totp.ts — TOTP / step-up second factor.
• platform-roles.ts, tenant-isolation.ts — role and tenant-boundary helpers used by the middleware.

The workspace-side identity client is libs/oshun/auth/, published as @oshun/auth-client, and RBAC / permission checks live in libs/shared/identity/.

The partial-failure envelope — a first-class contract#

When a write touches several owned records and some succeed while others fail, V1 does not throw away the successes or fabricate a clean 200. It returns a standardized partial-failure envelope, and that envelope is a real Zod contract at libs/contracts/src/common/partial-failure-envelope.ts — not just an informal shape.

The error element:

export const PartialFailureErrorSchema = z.object({
  domain: z.string().min(1),
  stage: z.string().min(1).optional(),
  message: z.string().min(1),
});

The envelope, with cross-field refinements that make the partial flag honest:

export const PartialFailureEnvelopeSchema = z
  .object({
    results: z.array(z.unknown()),
    errors: z.array(PartialFailureErrorSchema),
    partial: z.boolean(),
  })
  .superRefine((value, context) => {
    // partial=true  requires at least one error
    // partial=false requires an empty errors array
  });

The library also exports:

• createPartialFailureEnvelopeSchema(resultSchema, errorSchema?) — a generic factory that produces a typed envelope for any domain result type while re-applying the same partial ⇔ errors-non-empty refinement.
• buildPartialFailureEnvelope({ results, errors, partial? }) — a constructor that infers partial from the error count and throws if a caller passes a partial flag that contradicts the errors array, so a malformed envelope can't be built by hand.
• isPartialFailureEnvelope(value) — a runtime type guard.

Because the refinement is enforced in both directions, a service literally cannot claim partial: false while carrying errors, or partial: true with none — the contract refuses to lie about whether all the work succeeded.

Routing a write: how the foundations compose#

A customer- or operator-issued write threads the data-architecture pieces above in a fixed order before it reaches a domain adapter. The end-to-end sequence and its mermaid diagram live in Communication Patterns; the data-relevant beats:

1. Idempotency — libs/shared/http-client/src/idempotency.ts keys on the Idempotency-Key header (DEFAULT_IDEMPOTENCY_HEADER = 'Idempotency-Key'), caches for DEFAULT_IDEMPOTENCY_TTL_MS = 24h, and replays the stored envelope on a repeat. The same library also ships circuit-breaker.ts, retry.ts, ssrf-guard.ts, tenant-context.ts, and tracing.ts.
2. Tenant resolution reads X-Tenant-ID (or the token claim) and resolves the homeZone.
3. Residency (enforcer.ts) allows / denies / requires-consent against the home zone.
4. Zod validation parses the body against the domain contract.
5. Adapter commits owned data with tombstone semantics.
6. Event bus publishes a domain event (correlation-keyed) and the audit platform appends an immutable record.

The audit brief notes that the building blocks for this composition all exist and match the diagram; an end-to-end trace proving every middleware fires in exactly this order in production wiring was not performed. The ordering is the documented intent, grounded in the real middleware classes named above.

Each customer domain is reachable at a bff-base-path from the domain registry (libs/oshun/domain-registry/src/registry.ts). The six customer domains and their base paths:

OSHUN_DOMAIN_IDS enumerates these six; OSHUN_SHELL_PRIMARY_DOMAIN = 'tara' is the shell's default domain; DOMAIN_REGISTRY is the keyed metadata record each entry resolves to (display name, auth policy, admin taxonomy, deep-link prefix, launch contract, and more).

Correction: the event bus is not Redis Streams#

A recurring claim in the older architecture prose is that the event bus "runs over Redis Streams." It does not. libs/shared/event-bus/src/event-bus.ts exports class EventBus implements IEventBus with the factory createEventBus(config: EventBusConfig), and its persistence model — confirmed by reading the source — uses ioredis (import { Redis } from 'ioredis') with two separate connections (one publisher this.pub, one subscriber this.sub), built out of these primitives:

There is no XADD, XREAD, XREADGROUP, or XGROUP anywhere in the module — consumer groups are emulated with SET NX claim keys, not native Streams. The published package is @oshun/event-bus, and the same library also ships topic-registry.ts (schema-versioned topics), outbound-delivery.ts (outbound signing keys, retry/backoff, dead-letter inspection, replay, per-event audit), and webhook-simulator.ts (a dev-sandbox simulator for tenant integrations). Wherever a diagram participant or sentence still reads "Event Bus (Redis Streams)," read it as "the ioredis pub/sub, TTL-key, and sorted-set bus described here."

(Kafka remains the genuine Streams-class option, available behind the streaming Docker profile for high-volume cross-domain events when the default bus's capacity is exceeded.)

Background-job substrate — libs/shared/queue/#

Long-running editorial, asset, agentic-AI, notification, billing, and integration jobs all run on one shared, durable substrate at libs/shared/queue/src/:

• durable-queue.ts — the durable queue with priority classes, replay, and deduplication.
• dead-letter-queue.ts — the DLQ for jobs that exhaust retries.
• sla-monitor.ts — a per-job-class SLA monitor.
• memory-queue.ts — an in-memory implementation for tests / single-process dev.
• worker.ts — the worker loop; queue.ts / types.ts carry the shared interfaces.

Retry / dedup / replay / dead-letter contract tests are expected on every job submitter, so a job class can't ship without proving its replay semantics.

Inbound integration plumbing — libs/shared/inbound-integrations/#

Institutional and partner integrations land here. The src/index.ts re-exports fifteen connector modules — more than the older nine-row table surfaced:

* LOC shown for the three substantial connectors confirmed by wc -l; the others are real modules whose line counts weren't measured here.

† Fiat payment is V1.x optional — V1's primary entitlement path is non-custodial crypto via the Aje domain (libs/aje/) plus libs/oshun/payments-bridge/. See Support, Entitlements, Billing, and the Aje Entitlement Bridge.

The older docs' table mapped LMS → telemetry correctly but omitted the lti-verification, dual-SCORM (scorm-rte and scorm-2004-rte), byom-model, and calendar-google-transport siblings; the fifteen above are the real export set.

Identity, auth, and audit substrate#

• libs/shared/auth-primitives/ — credential, OAuth-client, TOTP/step-up, and tenant-isolation primitives (detailed above).

• libs/oshun/auth/ (@oshun/auth-client) — the workspace-side identity and session client every Oshun app consumes.

• libs/shared/identity/ — RBAC and permission checks.

• libs/shared/audit-platform/ — a large append-only audit substrate, not a one-liner. Its src/ holds roughly 40 implementation modules (≈89 files counting .spec/.test), including:

  • hash-chain.ts — a tamper-evident append-only hash chain. It exports hashEvent(), chainHashStep(), class HashChainedAuditEventStore (whose append() writes a chain entry alongside each event), verifyChain, an AuditChainTamperDetectedError, and createHashChainedAuditEventStore(). The chain entries live in an append-only sidecar; the underlying store can be Postgres, S3, or Kafka.
  • schema-versioning.ts, retention.ts, escalation-rules.ts, compliance-attestation.ts, watermark-verification.ts, synthetic-media-labeling.ts, voice-likeness-consent.ts, and provenance-bundle-validation (integration-tested).

  This substrate backs Iris admin inspection, DSAR review, the deletion workflow, and review packages. See Security, Privacy, and Compliance and Trust, Safety, and Privacy.

Contracts and codegen, briefly#

The data architecture is generated from contracts under libs/contracts/. libs/contracts/src/common/ contains 285 files (161 implementation .ts modules plus their .spec/.test siblings) — covering admin, billing, incident, persona, voice, persistence, and the partial-failure envelope above. (An older "≈265 files" figure is stale.)

libs/contracts/src/index.ts re-exports the common surface flat and uses namespace re-exports for collision-prone domains — more than the three the prose usually cites. The real namespaces include NisabaContracts, MetisContracts, VeritasContracts, and V3Contracts, V6Contracts, V9Contracts, and LivingSceneContracts, the last with explicit aliasing to avoid collisions (e.g. ScoreSchema as LivingSceneScoreSchema, deepParseScore as deepParseLivingSceneScore).

OpenAPI specs live at libs/openapi/src/specs/ across 15 spec directories — arete, bellona, calliope, concordia, hathor, isis, lilith, metis, nisaba, nyx, oshun-bff, sophia, tara, veritas, yemaya — plus top-level main.yaml and v3.yaml. Schemas drive the specs (Zod → OpenAPI), and a CI gate fails the build on spec drift against the runtime.

gRPC / Protocol Buffers are real and broader than the few domains the prose names. libs/proto/ carries buf.work.yaml, a generated/ tree, and ~24 proto domain directories under src/, including agent, ai, asset, auth, bridge, collaboration, concordia, generation3d, hathor, health, isis, loadbalancing, pipeline, procedural, project, reflection, rendering, shared, sophia, splatting, user, and common — used for internal service-to-service calls where latency or streaming matters.

Related#

• Communication Patterns — BFF request lifecycle, the event-bus topic table, and the Veritas retraction cascade.
• Foundations — the full foundation-library layer this page details.
• Security, Privacy, and Compliance — roles, scopes, step-up, and the audit substrate in context.
• Trust, Safety, and Privacy — DSAR, consent, and deletion as customer-facing surfaces.
• Iris — Assistant Memory Substrate — ConsentRecord and MemoryScope ownership.
• Customer-Facing Domains — the six domains behind the bff-base-path registry.
• Subsystem Glossary and the hub ../ARCHITECTURE.md.
• Backlog: §1.2 (tenancy contracts), §27 (Public API / OAuth), §28 (role model); see ../TODOS.md. Cross-domain dependencies: deps§1.