Status: Living document Date: 2026-04-27 Owner: eng-platform Backlog item: V1-P2-0371
This is the canonical view of how data, requests, and events flow across the
Oshun platform once the V1 Phase-2 foundational work has landed. It is the
"start here" doc for anyone joining a domain — it states what the moving parts
are, who owns each one, and what end-to-end flows look like in practice. The
root ARCHITECTURE.md is the reference for the V1
Phase-1 system as it shipped; this document is the delta — what changes once the
Phase-2 contracts (LLM gateway, embedding provider, cost-tracking, deterministic
mode, scanner conventions, external-API guard) are wired up.
Domains at a glance#
Six bounded contexts, each with its own data ownership, libraries, and event topic prefix:
| Domain | Topic prefix | Owns | Read-only consumers |
|---|---|---|---|
| Yemaya — creative studio | yemaya.* |
Projects, scripts, scenes, characters, shots, sequences, budgets, team membership | Lilith (course assets), Bellona (exports) |
| Lilith — consciousness experience | lilith.* |
Meditation sessions, courses, conversations, progress, biometrics, subscriptions | Sophia (knowledge queries), Tara (App-Store-safe variant) |
| Isis — generative factory | isis.* |
Generation jobs, assets, GPU workflows, provenance | Yemaya (asset feed), Lilith (meditation generation) |
| Sophia — research / knowledge | sophia.* |
Documents, embeddings index, citations, query history | Lilith (BFF knowledge route), Iris (RAG) |
| Hathor — worldbuilding | hathor.* |
Worlds, narrative lattices, NPC agents, simulation state | Yemaya (worldbuilding) |
| Bellona — build / integration | bellona.* |
Engine adapters, export packages, build jobs | Yemaya (export feed) |
In addition, three platform domains provide shared capabilities consumed by all six:
- Iris — embeddings, OCR, vision LLM, document layout analysis, desktop automation, A2A privacy primitives, semantic memory.
- Aja — motion capture, GPU motion pipeline, real-time pose / face / hand tracking. Consumed by Yemaya and Isis.
- Aje / Aphrodite / Airmid — cryptographic primitives (signing, key exchange, anonymization) consumed everywhere user PII or content provenance crosses a trust boundary.
Phase-2 foundational contracts#
These contracts are what changes between the Phase-1 system in
ARCHITECTURE.md and the Phase-2 end state. Every
domain consumes them; they bind the six contexts into one platform.
1. LLM gateway — IsisLLMClient#
Every prompt that crosses a process boundary flows through Isis. Phase-1 allowed
direct vendor calls (OpenAI, Anthropic, ComfyUI); Phase-2 requires
IsisLLMClient (V1-P2-0001 audit;
isis-llm-gateway-audit.md) so that:
- Cost accounting is centralized — every token is charged to a tenant via the
contract in
isis-cost-tracking-contract.md(V1-P2-0005). - Deterministic-mode replays work — the snapshot/replay semantics in
isis-deterministic-mode.md(V1-P2-0021) require the gateway to be the single point where requests are recorded and replayed. - Provider failover is uniform —
libs/iris/failover/src/failover-manager.tsprovides multi-provider orchestration that the gateway uses internally.
Caller (any domain)
│ Request: { tenantId, model, messages, traceId }
▼
IsisLLMClient (libs/isis/llm-providers)
│ ├── tenant quota check (cost-tracking-contract.md)
│ ├── deterministic-mode hook (isis-deterministic-mode.md)
│ ├── failover-manager + circuit breaker
│ └── provider call via @oshun/http-client (V1-P2-0358)
▼
Provider (OpenAI / Anthropic / local ONNX)
│ Response
▼
IsisLLMClient
│ ├── token accounting → bills to tenant
│ ├── trace span → otel collector
│ └── deterministic-mode write (when active)
▼
Caller receives { content, usage, costMicros, traceId }
2. Embedding gateway — IrisEmbeddingClient#
Sophia, Lilith (BFF knowledge), Iris (semantic memory), and Hathor
(narrative-lattice retrieval) all call IrisEmbeddingClient. The provider chain
is documented in
embeddings-provider-decision.md
(V1-P2-0030). The client wraps:
- The configured embedding provider (
bge-m3default, vendor fallback). - A request-level cache keyed by
(model, sha256(text)). - The same cost-tracking middleware as the LLM gateway, so embedding calls are billed alongside LLM calls.
- The same
@oshun/http-clientretry + circuit breaker + telemetry stack.
3. ML / on-device runtime#
Browser- and Node-side inference flows through the runtime and provider stack defined in V1-P2-021x:
ml-runtime-strategy.md— ONNX Runtime Web for browser, ONNX Runtime Node for server, WebGPU acceleration where available.pose-estimation-provider.md(V1-P2-0211),object-detection-provider.md(V1-P2-0212),asr-vad-provider.md(V1-P2-0213),speaker-id-provider.md(V1-P2-0214),emotion-detection-provider.md(V1-P2-0215),cross-encoder-reranker.md(V1-P2-0216).
For Aja motion capture in particular, the GPU-pipeline plan is in
aja-gpu-pipeline-strategy.md (V1-P2-0334)
and the Kubernetes deployment posture is in
aja-runtime-strategy.md (V1-P2-0333).
4. HTTP transport — @oshun/http-client#
Every outbound HTTP call uses libs/shared/http-client (createHttpClient /
HttpClient / withRetry / withCircuitBreaker / tracedRequest) or one of
the domain wrappers (libs/lilith/service-lib/circuit-breaker,
libs/iris/failover, libs/hathor/client, libs/sophia/client,
libs/bellona/client, libs/iris/agents/tools/web/http-client). The CI gate in
.github/workflows/external-api-guard.yml
(V1-P2-0358) refuses any new raw fetch( / axios( / got( / https.request(
call that lacks both retry and circuit breaker and telemetry.
5. Persistence — Drizzle migration#
libs/yemaya/database, libs/lilith/service-lib/database, the Isis, Sophia,
Hathor, and Bellona databases all run on PostgreSQL. The Phase-2 work tracked in
drizzle-completion-targets.md (V1-P2-0233)
walks every domain off placeholder persistence layers onto Drizzle ORM.
6. Cryptography — placeholder removal#
Every site flagged in
crypto-replacement-targets.md (V1-P2-0176)
— libs/aje/, libs/aphrodite/, libs/airmid/, libs/iris/memory/privacy/,
libs/iris/a2a/, libs/iris/privacy/security/dlp/ — moves from stubbed crypto
to the real primitive (@oshun/aje for signing/key-exchange, @oshun/aphrodite
for homomorphic operations, @oshun/airmid for differential privacy / noise
injection).
7. Event envelope contract#
Every cross-domain event uses the envelope from
ARCHITECTURE.md § Communication Patterns:
interface DomainEvent<T> {
id: string; // UUIDv7 (sortable)
type: string; // e.g. "isis.asset.generated"
source: string; // owning domain
timestamp: Date;
correlationId: string; // request-trace ID
payload: T;
metadata: Record<string, unknown>;
}
Producers commit owned data first, then publish. Subscribers idempotently key
side effects on event.id and store processed-event records in the subscriber's
own database. The Redis Streams transport is durable; failed subscribers are
replayed until the projection succeeds.
8. Quality / scanner contracts#
Phase-2 introduces seven CI workflows that gate or report on the §2.6 quality bar. They span every domain — see the README for the index.
| Workflow | Tickets | What it gates |
|---|---|---|
stub-indicator-scan |
V1-P2-0350 | New stub-vocabulary in PR diff |
silent-random-scan |
V1-P2-0351 | New Math.random() in deterministic functions |
test-coherence-check |
V1-P2-0352 | Per-file SHAPE/BOOLEAN/COMPUTED ratio (reporting) |
adversarial-grep-trend |
V1-P2-0355 | Nightly per-domain count + week-over-week delta (reporting) |
precommit-hook-verification |
V1-P2-0356 | Husky stub scan contract |
stub-indicator-pr-comment |
V1-P2-0357 | Sticky PR comment with file:line + remediation |
external-api-guard |
V1-P2-0358 | New raw HTTP calls without retry+CB+telemetry |
The // stub:legitimate <reason>, // random:legitimate <reason>, and
// http:guarded <reason> allow-list conventions are documented at
stub-indicator-allowlist.md and inline in
scripts/external-api-guard.mjs.
Canonical end-to-end flows#
These are the data flows the platform supports once the foundational work is complete. Each flow lists the originating user action, the participating domains, the events emitted, and the durability boundary.
Flow 1 — Yemaya generates an asset via Isis#
User in studio-web requests "generate concept art for shot 12"
│
▼
Yemaya BFF (apps/yemaya/api)
│ Validate: project membership, content-policy gate, budget remaining
│ Persist: GenerationRequest row in yemaya_db
│
▼ POST /isis/jobs (gRPC, mutual TLS)
Isis API (apps/isis/api)
│ Persist: Job row in isis_db
│ Enqueue: BullMQ "generation" queue, attempt budget = 3
│ Publish: isis.job.queued
│
▼ worker
Isis worker (apps/isis/workers)
│ Pull image-gen pipeline from libs/isis/pipelines
│ Call provider via IsisLLMClient OR ComfyUI (libs/isis/comfyui)
│ ├── @oshun/http-client retries on 5xx / network
│ ├── Circuit breaker skips dead providers (failover-manager)
│ ├── HttpTracing emits OTel span; cost-tracker bills tenant
│ Stream resulting image bytes to MinIO bucket isis-assets/
│ Persist: Asset row in isis_db with provenance metadata
│ Publish: isis.asset.generated
│
▼ Redis Streams subscriber
Yemaya event-handler (libs/yemaya/event-handlers)
│ Project: asset_link row in yemaya_db, key=event.id (idempotent)
│ Notify: WebSocket push to studio-web via svc-real-time-sync
▼
User sees asset in shot 12's gallery.
Durability boundary: the Isis Job row + MinIO object survive a crash mid-flow. Failed worker attempts are retried by BullMQ; failed Yemaya projections are replayed from the Redis Stream until the row is written.
Flow 2 — Lilith generates a personalized meditation#
User in Lilith app taps "today's meditation"
│
▼
Lilith BFF (apps/lilith/bff)
│ Resolve user profile + progress + biometrics from lilith_db
│ Resolve relevant teaching from Sophia knowledge graph:
│ ├── Sophia search uses IrisEmbeddingClient (V1-P2-0030)
│ └── Returns top-K passages with citations
│
▼ POST /isis/meditation (gRPC)
Isis svc-meditation-generation
│ IsisLLMClient generates script (cost-tracking-contract enforces quota)
│ Isis TTS gateway (V1-P2-0140) renders audio (provider chain)
│ Determinism: when seed is fixed, replay path stores prompt + audio
│ fingerprint per isis-deterministic-mode.md
│ Stream audio to MinIO bucket lilith-meditations/
│ Publish: isis.meditation.ready
│
▼ Redis Streams subscriber
Lilith svc-daily-content
│ Project: today_meditation row in lilith_db
│ Push to user via svc-notification (idempotent on event.id)
Flow 3 — Sophia ingests + indexes a document#
User uploads PDF to apps/sophia/api/ingest
│
▼ Sync intake
Sophia api persists Document row in sophia_db with status=ingesting
Sophia publishes sophia.document.intake on Redis Stream
│
▼ worker
Sophia ingestion worker
│ Document layout analysis: libs/iris/doc-layout (V1-P2-0066)
│ OCR pass for scanned pages: libs/iris/ocr (V1-P2-0060)
│ Vision LLM caption pass: libs/iris/vision-llm (V1-P2-0064)
│ Chunk + embed via IrisEmbeddingClient
│ Write: sophia.embeddings index (PostgreSQL pgvector or external store)
│ Update: Document row status=indexed
│ Publish: sophia.document.ingested + sophia.index.updated
│
▼ Subscribers
Lilith BFF (knowledge route) projects updated index pointer
Iris RAG cache invalidates per document_id
Flow 4 — Hathor simulates a worldbuilding tick#
Yemaya world editor calls Hathor "simulate next tick"
│
▼
Hathor api persists SimulationRequest in hathor_db
Hathor publishes hathor.simulation.requested
│
▼ worker
Hathor narrative-lattice engine
│ Pull world state from hathor_db
│ For each NPC: IsisLLMClient call shaped by lattice constraints
│ Detect contradictions, resolve via Sophia citation lookup
│ Persist: WorldState row + NPC actions in hathor_db
│ Publish: hathor.simulation.completed
│
▼ Subscribers
Yemaya world editor projects updated state
Sophia indexes new lore entries
Flow 5 — Bellona builds + exports#
Yemaya project lead clicks "export to Unreal 5"
│
▼
Yemaya api packages export bundle metadata, persists ExportRequest
Yemaya publishes yemaya.export.requested
│
▼ worker
Bellona engine adapter (libs/bellona/adapters/unreal)
│ Pull authoritative assets from Isis MinIO buckets
│ Transform to engine-specific format (FBX, USD, …)
│ Stream to bellona-exports/ MinIO bucket
│ Persist: Export row in bellona_db
│ Publish: bellona.export.ready
│
▼ Subscribers
Yemaya UI shows download link
Sophia indexes export manifest for citation
Shared infrastructure topology#
┌──────────────────────────────────────────────────────────────────┐
│ CLIENT LAYER │
│ studio-web · studio-desktop · iris-mobile · cli · sdks (ts/cpp/py) │
└─────────────────────────────────────────────────────┬────────────┘
│
┌────────────────────────────────────┴────────────────────────────────────┐
│ API GATEWAY (Hono + OpenAPI) │
│ JWT (auth-orchestrator) · rate-limit · @oshun/http-client │
└─────────────────────────────────┬────────────────────────────────────────┘
│
┌───────────────────┬──────────────────────┬┴──────────────────┬──────────────────┬───────────────────┐
▼ ▼ ▼ ▼ ▼ ▼
Yemaya Lilith Isis Sophia Hathor Bellona
(yemaya_db) (lilith_db) (isis_db) (sophia_db) (hathor_db) (bellona_db)
│ │ │ │ │ │
└─────┬─────────────┴──────────────────────┴───────────────────┴──────────────────┴───────────────────┘
│
┌─────────┴────────────────────────────────────────────────────────────────────────────────────────────┐
│ SHARED INFRASTRUCTURE │
│ │
│ Redis Streams + Pub/Sub (event bus) │
│ Redis (cache / rate-limit buckets) │
│ BullMQ on Redis (job queues per domain) │
│ PostgreSQL (one DB per domain — Drizzle ORM, V1-P2-0233) │
│ pgvector (Sophia embeddings index; optional Qdrant via observability profile) │
│ MinIO (asset object store: yemaya-assets/, isis-assets/, lilith-meditations/, bellona-exports/) │
│ Vault / VaultClient (secrets, with retry + cache; libs/lilith/service-lib/vault-client.ts) │
│ OpenTelemetry collector → Prometheus + Tempo + Loki │
│ dashboards live at docker/observability/grafana/dashboards/<domain>/ │
│ Mailpit (dev) / SES (prod) (notifications) │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘
Per-domain database isolation is enforced at the schema level — domains do not query each other's tables. Cross-domain reads happen via API or via projected read-models written by event-handler subscribers.
Trust boundaries#
The Phase-2 system has three trust boundaries that gate every flow:
- External user → API gateway. JWT auth, content-policy filtering, rate
limit per tenant. The gateway is also where
@oshun/http-clientis wired up — every outbound provider call from a domain goes through the same retry + circuit-breaker + telemetry stack as the inbound request. - Domain → Vault. No domain stores raw vendor API keys; every key is
fetched via
VaultClient.getSecret()with the 5-minute in-memory cache + 3x retry. Per V1-P2-0358 the vault client itself is on the wrapper allow-list (it's a primitive; it can callhttps.requestdirectly). - Domain → Domain. mTLS gRPC for synchronous call-outs; signed event
envelopes for asynchronous broadcast. Subscribers verify the event's
sourceand idempotently apply byevent.id.
Observability surface#
Every flow emits the same three signal types:
- Logs —
pinostructured JSON, scoped bycorrelationId==event.correlationIdso a single user action can be reconstructed end to end across domain boundaries. - Metrics — Prometheus scrape on every
apps/*/apiservice. Per-domain dashboards underdocker/observability/grafana/dashboards/<domain>/enumerated in the README. - Traces — OpenTelemetry OTLP via the
HttpTracinginterceptor in@oshun/http-client. Tempo retains 30 days; Grafanaoshun/platform-overview.jsonis the top-of-funnel view.
For per-subsystem operational reference, every domain owns a runbook generated from the Runbook Template (V1-P2-0372), and every Tier-A subsystem owns a threat model generated from the Threat Model Template (V1-P2-0373).
Differences from ARCHITECTURE.md#
This document is the Phase-2 view; the table below highlights what changes from
the Phase-1 baseline in ARCHITECTURE.md:
| Concern | Phase-1 baseline | Phase-2 contract |
|---|---|---|
| LLM provider | Direct vendor SDK calls scattered across domains | Centralized IsisLLMClient (V1-P2-0001), cost-tracked, deterministic-replayable |
| Embeddings | Per-domain vendor calls | IrisEmbeddingClient (V1-P2-0030) |
| HTTP transport | Mix of raw fetch/axios and ad-hoc wrappers |
@oshun/http-client everywhere, gated by external-api-guard (V1-P2-0358) |
| Crypto | Stub primitives in privacy/security paths | Real primitives per crypto-replacement-targets.md (V1-P2-0176) |
| Persistence | Mixed (some Drizzle, some placeholder) | Uniform Drizzle (V1-P2-0233) |
| Quality bar | Manual review | 7 CI workflows enforce §2.6 (V1-P2-0350..0358) |
| Documentation | Free-form per domain | Templated docs under docs/releases/p2/ (V1-P2-0370..0373) |
When the items in the right column have all moved from [ ] to [x] in
OSHUN_V1_TODOS_P2.md, this document IS the
operating architecture; any divergence in code is a defect to fix, not a
documentation update to make.