Disciplines · Proposals

Infrastructure Inventory Available to the Workspace (YSD-0105)

Domain-isolated databases provisioned by

12sections4 minread
Markdown source
On this page

Compiled 2026-07-18 at working-tree commit 84c7410039 from docker/docker-compose.dev.yml, infra/, .env.example, and the YSD-0104 schema sweep. Complements the schema inventory (what data looks like) with what stores/systems exist to hold and move it.

Databases#

  • PostgreSQL 16 + pgvector (dev: pgvector/pgvector:pg16, container oshun-dev-postgres, via PgBouncer oshun-dev-pgbouncer). Domain-isolated databases provisioned by docker/scripts/init-multiple-databases.sh from POSTGRES_MULTIPLE_DATABASES: yemaya, lilith, isis, iris, sophia, hathor, bellona, calliope, tara, maat, nisaba, shakti, cybele, kalika, lakshmi, athena, oya, phoebe plus the default oshun_dev. Both yemaya and nisaba databases already exist — the workspace's authoritative stores (Section 3.1) have homes without new provisioning.
  • ORM landscape (YSD-0104): Prisma schemas + migrations for nisaba, sophia, yemaya, oshun (isis has generated Prisma client models); metis uses a drizzle-style schema module materialized in oshun; yemaya/assets has a drizzle SQL migration.
  • Prod: Terraform under infra/terraform/ (post 2026-07-04 EKS→ECS-Fargate migration): vpc, ecs, ecs-services, ecs-roles, alb, ecr, secrets, ssm, cloudwatch, state-backend, github-oidc, per-domain stacks (iris, psyche, tara), environments/.

Object stores#

  • MinIO (dev: minio/minio:latest + minio-init bucket bootstrap), S3-compatible — maps to Section 3.2 rights-aware object storage.
  • S3 (prod: infra/terraform/s3/).
  • Existing S3-pattern consumer: libs/shared/live-media/src/ s3-media-object-store.ts (+ spec) — the closest live example of an object store wrapper for media, added 2026-07-18.
  • Git-LFS quota accounting for large assets: libs/yemaya/assets/drizzle/.

Queues, jobs, and events#

  • Redis 7 (dev redis:7-alpine) with BullMQ via the shared wrapper libs/shared/queue (BullMQ dependency; also used by veritas/themis/nous/ lakshmi) — default jobs substrate.
  • SQS + DLQs (prod: infra/terraform/sqs/ with per-queue dead-letter queues, FIFO support, depth alarms; queue URLs exported as <NAME>_QUEUE_URL), SNS (infra/terraform/sns/).
  • Kafka 7.5 + ZooKeeper (dev, streaming profile) and Debezium CDC (docker/debezium/) — available for event streaming/outbox capture but profile-gated, not default.
  • YSD-3045's transactional-outbox requirement has no ready-made shared implementation; Debezium or a Postgres outbox table + relay are the candidate mechanisms.

Search indexes#

  • Elasticsearch 8.11 (dev, search profile; prod-side config also under infra/yemaya/elasticsearch + kibana, and infra/elk).
  • PostgreSQL FTS available by default in every domain database.
  • Existing search services: apps/sophia/search-api (BM25/hybrid/rerank vocabulary per YSD-0104), libs/sophia/semantic-search.

Vector stores#

  • pgvector — default, in every domain database (dev image is pgvector-enabled).
  • Qdrant (dev, vectors profile) — dedicated vector DB when scale justifies.
  • Existing embedding schemas: aja VideoEmbedding, sophia semantic-search types.

Graph projections#

  • Neo4j 5.26 community (dev, graph profile) — available for the Section 3.3 relationship-graph projection when measured need justifies it; nothing currently forces a graph DB (sophia's knowledge-graph service and yemaya's canon-graph are TypeScript/Postgres-backed).
  • Existing graph engines (YSD-0104): libs/sophia/knowledge-graph, libs/sophia/citation-graph, libs/yemaya/canon-graph, nisaba InfluenceEdge/ConceptMapping tables.

Time-series stores#

  • None dedicated (no Timescale/ClickHouse/InfluxDB anywhere in dev compose or terraform). Prometheus (dev observability profile; prod infra/monitoring, infra/yemaya/grafana) covers operational metrics only. This matches YSD-3043's posture: build game-session/event time-series storage only where measured volume justifies it; V2's telemetry apps currently model schema/migration/replay in TypeScript without a dedicated TSDB.

Audit systems#

  • Canonical contract: libs/contracts/src/common/canonical-audit-event.ts (CanonicalPlatformAuditEventSchema, severity/actor/target/evidence/ outcome).
  • Hash-chained store + ingest: libs/shared/audit-platform (HashChainedAuditEventStore, chainHashStep, AuditEventIngestService; InMemoryCanonicalAuditEventStore is the only bundled store — a durable Postgres-backed store is required before YSD-3060 can claim tamper-evident production audit).
  • Domain audit tables: sophia AuditLog, yemaya AuditLog, oshun AdminAuditEvent/OshunV1TombstoneAuditEvent/RailRingOptInAuditEvent (divergent shapes — conflict #8 in the schema inventory).
  • Second hash chain: bellona RemoteAuditChainExport (replay-tied).

Observability#

  • Dev observability profile: Prometheus v2.47, Grafana 10.2, Jaeger 1.51 (traces). Prod: infra/monitoring, CloudWatch (terraform), infra/elk.

GPU / model execution#

  • No on-box GPU (dev-box toolchains memory). RunPod endpoints (infra/runpod/endpoints) are the GPU execution path post open-model migration; isis gpu-worker targets them. Workspace analyzers must treat GPU-dependent lenses as capability-gated (YSD-2007) rather than assuming local execution.

Dev conveniences#

  • Mailpit (SMTP capture), tools profile: redis-commander, pgAdmin, kafka-ui. Test defaults are safe (localhost-only credentials in .env.example; no root .env committed).

Gaps the workspace must fill (feeding Sections 1, 3, 6)#

  1. No transactional outbox implementation (YSD-3045) — candidates: Postgres outbox + BullMQ relay, or Debezium CDC.
  2. No durable store behind the hash-chained audit ledger (YSD-3060).
  3. No lifecycle/quarantine bucket conventions for study media (YSD-3020) — live-media's publication prefixes are the nearest pattern.
  4. Elasticsearch/Qdrant/Neo4j/Kafka are profile-gated: local development of search/vector/graph projections must degrade to pgvector/Postgres-FTS-only mode or document the profile requirement (YSD-1011).