# Foundations The foundation layer is the shared substrate every V1 surface stands on: the contracts that define each object once, the persistence machinery that keeps those contracts and the database in lockstep, the BFF middleware chain that guards every write, and the queue/event/integration/identity plumbing that makes the [customer-facing domains](./customer-domains.md) and [product surfaces](./product-surfaces.md) interoperable. It serves engineers across every domain — none of them re-implement idempotency, residency, audit, or eventing — and it sits underneath the [High-Level Architecture](./high-level-architecture.md), between the domain adapters and the raw infrastructure. The work is tracked in [TODOS § 1](../TODOS.md#1-foundations-contracts-adapters-persistence), and almost every piece below is real code with tests already in the repo. Where something is contract-level rather than a live deployment, this page says so. > **Status candor.** This area is overwhelmingly _implemented_, not > aspirational. Contracts, the persistence drift/index/migration/tombstone test > quartet, the OAuth 2.1/PKCE module, the role/scope model, the partial-failure > envelope contract, the durable queue + DLQ + SLA monitor, the topic-registry + > outbound-delivery + webhook-simulator event bus, the inbound connectors, the > data-residency enforcer, the idempotency middleware, and the audit-platform > hash chain all exist with tests. The two honestly-scoped caveats: the OAuth > module is **contracts/state-machine level** (token types, PKCE validation > patterns, refresh/revocation surface) and does not stand up a live > authorization-server deployment. The BFF request-lifecycle diagram below > describes a real composition of building blocks that are all present, but no > single production route was traced end-to-end proving every middleware fires > in exactly that order. --- ## 1. Contracts — `libs/contracts/` V1 holds to **one source of truth per object class**. Each contract is a Zod schema with a contract-test fixture that round-trips a known-good payload, so a schema change that breaks an existing object fails CI rather than silently shipping. - **Common contracts** live under `libs/contracts/src/common/`. That directory now contains **285 files** — **161 implementation `.ts` modules** plus their co-located `.spec.ts`/`.test.ts` siblings — covering admin, billing, incident, persona, voice, persistence, and many more cross-cutting object classes. (Earlier drafts of this doc cited "≈265 files," which was stale and low.) - **Domain contracts** live under `libs/contracts/src//`. - Everything re-exports through `libs/contracts/src/index.ts` as `@oshun/contracts`. To avoid naming collisions across families, several domains are exported as **namespaces** rather than flat symbols. The index uses `export * as` for `NisabaContracts` (`./nisaba`), `MetisContracts` (`./metis`), and `VeritasContracts` (`./veritas`) — beyond the three customer domains, also for **`V3Contracts`** (`./v3/index`), **`V6Contracts`** (`./v6/index`), **`V9Contracts`** (`./v9/index`), and **`LivingSceneContracts`** (`./living-scene/index`). Where a generic name would collide, the index re-exports under an explicit alias — e.g. `ScoreSchema as LivingSceneScoreSchema` and `deepParseScore as deepParseLivingSceneScore` — so consumers can import a Living Scene score without shadowing another domain's `ScoreSchema`. - **Specialized contract packages** exist for Veritas, Psyche, Iris, Maat, Cybele, Concordia, and Saraswati when a consumer needs a smaller surface than the umbrella package. ### The partial-failure envelope contract The standardized partial-failure envelope (§5, below) is not a convention — it is a **first-class Zod contract** at `libs/contracts/src/common/partial-failure-envelope.ts`. Its exported surface: | Symbol | What it is | | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `PartialFailureEnvelopeSchema` | The base schema: `{ results: unknown[], errors: PartialFailureError[], partial: boolean }`. | | `PartialFailureErrorSchema` / `PartialFailureError` | A single failure: `{ domain: string, stage?: string, message: string }` (each string non-empty; `stage` optional). | | `createPartialFailureEnvelopeSchema(resultSchema, errorSchema?)` | Generic factory that produces a typed envelope schema for a specific result/error shape, defaulting the error schema to `PartialFailureErrorSchema`. | | `buildPartialFailureEnvelope({ results, errors, partial? })` | Constructs a well-formed envelope; throws if a supplied `partial` flag disagrees with whether `errors` is non-empty. | | `isPartialFailureEnvelope(value)` | Structural type guard. | The base schema's cross-field refinement is what makes it trustworthy: a `superRefine` enforces that **`partial: true` requires at least one error** and **`partial: false` requires an empty `errors` array**. An envelope that claims to be partial but carries no errors — or claims success while carrying errors — is rejected at parse time: ```ts PartialFailureEnvelopeSchema.parse({ results: [ /* 8 ok */ ], errors: [{ domain: 'metis', stage: 'grading', message: 'timeout' }], partial: true, // valid: partial=true ⇒ errors non-empty }); ``` This is why the BFF can normalize a fan-out across several domains into one response that is honest about which slices succeeded — the contract makes a lie unrepresentable. --- ## 2. Adapters — `libs/oshun/domain-*` and substrate adapters - Every customer-facing domain exports one **canonical adapter** (the typed read APIs the shared shell, admin, and assistant consume) and one **adapter** variant for boundary-specific concerns (BFF routes, fixtures). - **Substrate adapters** present the same shape regardless of whether the underlying substrate is local-only or distributed, so callers don't branch on deployment topology. - Adapters are wired through the BFF via the **domain registry**. The registry at `libs/oshun/domain-registry/src/registry.ts` exports `OSHUN_DOMAIN_IDS` and `DOMAIN_REGISTRY`, and pins the shell's primary domain via `OSHUN_SHELL_PRIMARY_DOMAIN = 'tara'`. Each of the six customer domains carries a `'bff-base-path'`: | Domain | `bff-base-path` | | ------- | ---------------------------- | | Tara | `/api/oshun/domains/tara` | | Veritas | `/api/oshun/domains/veritas` | | Nyx | `/api/oshun/domains/nyx` | | Arete | `/api/oshun/domains/arete` | | Nisaba | `/api/oshun/domains/nisaba` | | Metis | `/api/oshun/domains/metis` | See the [Subsystem Glossary](./glossary.md) and [Customer-Facing Domains](./customer-domains.md) for what each domain owns. ### The platform-foundations package Several of these adapter-adjacent concerns are not scattered across the tree — they have a single home. `libs/oshun/platform-foundations/` (`@oshun/platform-foundations`) re-exports **exactly nine subsystems** from its `src/index.ts`: | Subsystem | Concern | | ------------------- | -------------------------------------------------------- | | `service-discovery` | Resolving and addressing domain services. | | `public-api` | The §27 Public API platform, incl. OAuth 2.1/PKCE (§10). | | `shared-contracts` | Cross-subsystem contract surface. | | `role-model` | Canonical roles, scopes, field filtering (§10). | | `step-up` | Step-up authentication for sensitive actions. | | `secrets` | Secret handling. | | `configs` | Configuration. | | `rollback` | Rollback workflows. | | `abuse-controls` | Abuse/throttling controls. | --- ## 3. Persistence — `libs/oshun/persistence/` The persistence layer's job is to keep Zod contracts and the database from drifting apart, and to make user-data deletion both safe and auditable. - **Schema rendering.** `prisma-renderer.ts` and `zod-prisma-introspection.ts` drive Prisma schema from Zod, not the reverse, so the contract remains the single source of truth. - **Drift test.** `contract-persistence-registry.test.ts` (backed by `contract-persistence-registry.ts`) fails on field divergence between the Zod contract and the persistence layer — a renamed or dropped field is a red build, not a runtime surprise. - **Tombstone semantics.** Mandatory on every user-data table: deletion is a **soft tombstone** that propagates and is audit-logged; there is no silent re-creation. Covered by `tombstone-semantics.test.ts`. - **Required indexes.** `index-requirements.test.ts` validates per-table indexes via `EXPLAIN` checks in CI, so an unindexed hot path can't land quietly. - **Migration plans.** `migration-plan.test.ts` keeps migrations idempotent and dry-runnable. Beyond the core quartet, the same library houses the **DSAR deletion machinery** — `dsar-deletion-cascade.ts` (with both unit and `.integration` tests) and `dsar-erasure-runtime.ts` — which is how a privacy-operator's erasure request fans the tombstone across every owning table. This is the persistence-side counterpart to the [Trust, Safety, and Privacy](./trust-safety-and-privacy.md) workflows. --- ## 4. OpenAPI and codegen — `libs/openapi/` - Specs live under `libs/openapi/src/specs/`. There are **15 real spec directories**: `arete`, `bellona`, `calliope`, `concordia`, `hathor`, `isis`, `lilith`, `metis`, `nisaba`, `nyx`, `oshun-bff`, `sophia`, `tara`, `veritas`, and `yemaya` — plus top-level `main.yaml` and `v3.yaml`. - **Schemas drive specs** (Zod → OpenAPI), never the reverse, so the contract layer remains authoritative. - Typed clients are generated per domain into `libs//api-client/`. - **CI gate:** spec drift against runtime fails the build. --- ## 5. Idempotency, partial-failure, response normalization - **Idempotency middleware** lives in `libs/shared/http-client/src/idempotency.ts`. It defaults to the `Idempotency-Key` header (`DEFAULT_IDEMPOTENCY_HEADER`) with a 24-hour replay window (`DEFAULT_IDEMPOTENCY_TTL_MS = 24 * 60 * 60 * 1000`). The `IdempotencyMiddleware` class — constructed via `createIdempotencyMiddleware()` — gives request deduplication and replay-safe semantics: a repeated key returns the stored envelope instead of re-executing the write. - The same `libs/shared/http-client/` package ships the rest of the outbound hardening: `circuit-breaker.ts`, `ssrf-guard.ts`, `retry.ts`, `tenant-context.ts`, and `tracing.ts`. So an outbound call from the BFF to a domain (or to an external connector) is idempotent, breaker-protected, SSRF-guarded, retried, tenant-scoped, and traced through one composed client. - **Partial-failure envelope.** The standardized shape `{ results[], errors[], partial }` is the §1 contract above — used across the BFF and domain services so a multi-domain fan-out can report exactly which slices failed without collapsing into a single opaque error. - **Job contracts.** Retry/dedup/replay/dead-letter contract tests run on every job submitter. --- ## 6. Multi-tenant routing — `libs/shared/data-residency/` - Tenant-aware middleware reads `X-Tenant-ID` (or the token claim) and propagates the tenant context through every downstream call. - **Residency-aware traffic shaping.** `traffic-shaping.ts`, `home-zone.ts`, and `enforcer.ts` route a request to the user's primary (home-zone) data plane unless explicit cross-region consent is present. `dsr-routing.ts` handles data-subject-request routing, and an injected `ResidencyAuditPublisher` seam (exercised by `audit-platform-bridge.spec.ts`) ties residency decisions into the audit substrate (§10). - **Tenant-aware caching keys** prevent cross-tenant cache pollution. - **Tenant-aware feature flags** and experiment scoping. - Cross-tenant leakage tests live under `tests/security/tenant-isolation/`. This is the enforcement spine behind [Data Architecture and Tenancy](./data-architecture-tenancy.md) and the residency obligations in [Security, Privacy, and Compliance](./security-privacy-compliance.md). --- ## 7. Background-job substrate — `libs/shared/queue/` A single durable queue is shared by editorial, asset, agentic-AI, notification, billing, and integration jobs — there is not a bespoke queue per domain. - `durable-queue.ts` — priority classes, replay, deduplication, observability hooks. - `dead-letter-queue.ts` — the DLQ for jobs that exhaust retries. - `sla-monitor.ts` — a per-job-class SLA monitor. - `memory-queue.ts` — an in-memory implementation for tests/local development. - `worker.ts` — the worker loop that drains the queue. --- ## 8. Webhook and event-bus plumbing — `libs/shared/event-bus/` The event bus is `@oshun/event-bus`; its surface is `export class EventBus implements IEventBus` with a `createEventBus(config: EventBusConfig)` factory. Alongside it: - `topic-registry.ts` — topics with schema versioning (`DEFAULT_EVENT_TOPIC_REGISTRY`), so a payload that doesn't match the registered topic schema is rejected before fan-out. - `outbound-delivery.ts` — outbound signing keys, retry/backoff, dead-letter inspection, replay, and per-event audit for tenant-facing webhooks. - `webhook-simulator.ts` — a simulator tenant integrations can use in a dev sandbox. ### How the bus actually works — it is **not** Redis Streams Earlier drafts (and the cascade diagram below) labeled the bus "over Redis Streams." That is the one outright-wrong technical claim, and it's worth correcting precisely, because the real design is more interesting. The implementation imports `import { Redis } from 'ioredis'` and opens **two separate connections** — `this.pub` and `this.sub` — but it uses **none** of the native Streams commands (no `XADD`, `XREAD`, `XREADGROUP`, or `XGROUP`). Instead it composes Streams-like guarantees out of simpler primitives: | Concern | Mechanism | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Fan-out | Redis **pub/sub** — `this.pub.publish(channel, …)` out, pattern-subscribe (`pmessage`) in. | | Replay source | A **TTL-bounded key** per event (`setex`, default `eventTtl` = 24h), so a crash-restart can re-deliver unacked work. | | Delay / nack | A durable **sorted set** named `scheduled` (`zadd(scheduledKey, deliverAt, …)`); a scheduler loop (default 250 ms) pulls due entries. `setTimeout` state is never the only copy. | | Consumer groups | A per-`(eventId, group)` **`SET … NX`** claim key (`this.pub.set(key, '1', 'EX', eventTtl, 'NX')`): only the winner runs the handler. Without a group, every matching subscription runs (classic broadcast). | | Dead letter | A **list** (ordered pagination) plus a **hash** (O(1) id lookup); `removeDeadLetter` uses `LREM` for atomic removal. | So the bus is genuinely durable and consumer-group-capable, but via pub/sub + TTL keys + a sorted set + `SET NX` claims — not `XADD`/`XREADGROUP`. Treat any remaining "Redis Streams" wording in the hub doc as a known label error. --- ## 9. Inbound integration plumbing — `libs/shared/inbound-integrations/` The package's `src/index.ts` re-exports **15 connector modules** — more than the nine the original table listed. Several are substantial real implementations (e.g. `oneroster.ts` ≈1081 LOC, `lms.ts` ≈1425 LOC, `identity.ts` ≈1048 LOC), not thin wrappers. | Module | Connector family / role | V1 use | | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `types.ts` | Shared types | Connector contracts used by all the modules below. | | `lms.ts` | LMS | LTI 1.3, LTI Advantage, SCORM fallback for Metis institutions. | | `lti-verification.ts` | LMS | LTI launch/credential verification, split out from `lms.ts`. | | `scorm-rte.ts` | LMS | SCORM run-time environment (legacy). | | `scorm-2004-rte.ts` | LMS | SCORM 2004 run-time environment (second, distinct RTE). | | `oneroster.ts` | OneRoster | Rostering sync with conflict reporting and dry-run. | | `identity.ts` | Identity | SAML 2.0, OIDC, SCIM 2.0 for tenant SSO. | | `calendar.ts` | Calendar | Google/Apple/Outlook two-way sync for Tara/Arete/Nyx/Metis. | | `calendar-google-transport.ts` | Calendar | Google-specific transport for the calendar connector. | | `payment.ts` | Payment (fiat) | Fiat-rail entitlement upgrades (Stripe-class; Telegram payments). V1.x optional — the V1 primary path is non-custodial crypto via the **Aje** domain (`libs/aje/`) plus `libs/oshun/payments-bridge/`. | | `telemetry.ts` | Telemetry | xAPI/cmi5/Caliper export to institutional sinks. | | `byom.ts` | BYOM | Tenant-provided model endpoints for Metis study. | | `byom-model.ts` | BYOM | The model abstraction backing `byom.ts`, exported separately. | | `notification.ts` | Notification | Slack/Teams notification sinks for institutional surfaces. | | `health.ts` | Health | Connector health probes, circuit breakers, version pinning. | See [Messaging Channels](./messaging-channels.md) and the Aje bridge in [Support, Entitlements, Billing, and the Aje Entitlement Bridge](./support-billing-and-crypto.md) for how these surface to customers. --- ## 10. Identity, auth, and audit ### Auth primitives — `libs/shared/auth-primitives/` Richer than "JWT and session primitives." The package ships, among others: `jwt.ts`, `session.ts`, `api-key.ts`, `password.ts`, `oauth-client.ts`, `oauth-revoke.ts`, `token-refresh.ts`, `token-audit.ts`, `totp.ts` (TOTP for step-up), `platform-roles.ts`, and `tenant-isolation.ts` — each with a co-located `.spec.ts`. So API keys, password hashing, OAuth client + revocation, refresh rotation, token auditing, TOTP step-up, and tenant isolation are all primitives, not bespoke per-app code. ### Workspace identity client — `libs/oshun/auth/` `@oshun/auth-client` is the workspace-side identity and session client every Oshun app consumes. ### The role/scope model — `platform-foundations/.../role-model/` `libs/oshun/platform-foundations/src/role-model/role-model.ts` defines the **ten** `CANONICAL_ROLES` — note the precise names; prose elsewhere uses looser labels like "support" that don't match the code: | Canonical role | Scopes granted (`scopesFor(role)`) | | ------------------ | ------------------------------------------------------------------------------------ | | `customer` | `customer.self`, `customer.shell` | | `creator` | `customer.self`, `customer.shell`, `creator.studio` | | `support-agent` | `support.case.read`, `support.case.write` | | `reviewer` | `review.queue.read`, `review.queue.decide` | | `moderator` | `review.queue.read`, `review.queue.decide`, `moderation.decide` | | `privacy-operator` | `privacy.dsar.execute`, `privacy.audit.read`, `admin.audit.global` | | `model-operator` | `model.registry.read`, `model.registry.promote` | | `persona-operator` | `persona.registry.read`, `persona.registry.publish` | | `tenant-admin` | `tenant.console`, `admin.audit.global` | | `admin-leadership` | `admin.users.act`, `admin.scope.grant`, `admin.breakglass.act`, `admin.audit.global` | The full `SCOPE_KEYS` list (21 scopes) is the vocabulary `authorizeAction()`, `authorizeObjectAccess()`, and `filterFieldsByRole()` enforce. `creator` and `tenant-admin` are distinct canonical roles — a fact older role prose omits. The support role is `support-agent`, not `support`. Use these exact strings when wiring RBAC. ### OAuth 2.1 / PKCE — `platform-foundations/.../public-api/oauth.ts` The §27 Public API platform's OAuth module is **contracts/state-machine level**: it validates and computes, but does not host a running authorization server. Concretely it exports `OAUTH_GRANT_TYPES`, `TOKEN_TYPES = ['access', 'refresh']`, and a PKCE validator built on `PKCE_CODE_CHALLENGE_PATTERN = /^[A-Za-z0-9_-]{43,128}$/` (the RFC 7636 URL-safe-Base64 challenge length window). Its functions cover the real surface: `validateAuthorizationCodeRequest`, `validateAuthorizationCodeRedeem`, `verifyPkceChallenge`, `validateAccessToken`, `evaluatePublicApiQuota`, `buildPublicApiAuditEvent`, `redeemRefreshToken`, and `revocationCascade`. Error codes include `'pkce-required'` (authorization code request with no challenge) and `'pkce-verification-failed'` (redeem whose verifier doesn't match). PKCE validation patterns and the refresh/revocation surface are confirmed; a live authorization-server deployment is not part of this module. ### RBAC checks — `libs/shared/identity/` RBAC and permission checks are consumed across domains. ### Audit substrate — `libs/shared/audit-platform/` Far more than one sentence's worth: `libs/shared/audit-platform/` is a large real substrate (~40 implementation modules plus tests). At its core is `hash-chain.ts`, an **append-only hash chain**, so an audit record cannot be silently altered or deleted without breaking the chain. Around it: `schema-versioning.ts`, `retention.ts`, `escalation-rules.ts`, `support-escalation.ts`, `compliance-attestation.ts`, `watermark-verification.ts`, `synthetic-media-labeling.ts`, `voice-likeness-consent.ts`, and provenance-bundle validation (with `.integration` specs). This is the substrate behind Iris admin inspection, DSAR review, the deletion workflow, review packages, synthetic-media labeling, and voice-likeness consent — the trust-and-safety record of last resort. --- ## How the foundation composes — the BFF request lifecycle Every customer- or operator-issued write passes the same middleware sequence before reaching a domain adapter. Idempotency is checked first (replays return the cached envelope); tenant and residency resolve next so the rest of the pipeline can short-circuit on policy. The body is validated against the Zod contract; the adapter commits owned data with tombstone semantics; an event is published and an audit record appended. Each station maps to a real building block above — idempotency → `http-client/idempotency.ts`; tenant + residency → `data-residency/`; validation → `@oshun/contracts`; adapter → the domain adapters keyed by `DOMAIN_REGISTRY`; event → `@oshun/event-bus`; audit → `audit-platform/hash-chain.ts`. (The ordering shown is the intended composition; no single production route was traced proving every middleware fires in exactly this order.) ```mermaid sequenceDiagram autonumber actor C as Client participant BFF as Oshun BFF participant Idem as Idempotency Cache participant T as Tenant Middleware participant R as Residency Enforcer participant V as Zod Contract Validator participant D as Domain Adapter participant DB as Domain DB participant EB as Event Bus participant A as Audit Platform C->>BFF: POST /api/oshun/domains/X
Idempotency-Key · X-Tenant-ID · JWT BFF->>Idem: lookup(idempotencyKey) alt cache hit Idem-->>BFF: stored response BFF-->>C: 200 OK (replayed) else cache miss BFF->>T: resolve tenant + scopes T-->>BFF: tenant context · homeZone BFF->>R: check residency vs request region R-->>BFF: allow / require consent BFF->>V: validate body V-->>BFF: parsed payload BFF->>D: invoke(operation, payload) D->>DB: write owned data (tombstone-aware) DB-->>D: ack D->>EB: publish domain event
{correlationId} D->>A: append mutation record D-->>BFF: result BFF->>Idem: store(key, response) BFF-->>C: 200 OK end ``` A partial fan-out at the BFF — say one operation that touches three domains — collapses into the §1 partial-failure envelope rather than a single opaque error, so the client learns exactly which slice failed. The asynchronous side and the Veritas retraction-cascade walkthrough live on the dedicated [Communication Patterns](./communication-patterns.md) page (which also inherits the corrected "not Redis Streams" framing from §8). For internal service-to-service calls, the gRPC layer is real too: `libs/proto/` carries `buf.work.yaml`, a `generated/` tree, and ~24 proto domain directories under `src/` — `agent`, `ai`, `asset`, `auth`, `bridge`, `collaboration`, `concordia`, `generation3d`, `hathor`, `health`, `isis`, `loadbalancing`, `oshun`, `oya`, `pipeline`, `procedural`, `project`, `reflection`, `rendering`, `shared`, `sophia`, `splatting`, `user`, and `common` — well beyond the Psyche/Isis/Sophia trio commonly cited. --- ## Related - [High-Level Architecture](./high-level-architecture.md) — where the foundation sits. - [Communication Patterns](./communication-patterns.md) — the sync/async/gRPC layer built on §7–§8. - [Data Architecture and Tenancy](./data-architecture-tenancy.md) — the residency/persistence story extended. - [Customer-Facing Domains](./customer-domains.md) — the domains the adapters (§2) front. - [Subsystem Glossary](./glossary.md) — names and ownership. - [Security, Privacy, and Compliance](./security-privacy-compliance.md) — RBAC, residency, and audit obligations. - [Trust, Safety, and Privacy](./trust-safety-and-privacy.md) — DSAR, tombstones, and the audit substrate in use. - Hub: [../ARCHITECTURE.md](../ARCHITECTURE.md)