# Isis — Architecture > Technical architecture for the Generative AI Factory domain. --- Isis is the generative AI content factory at the heart of the Oshun platform. Its job is to accept generation requests from any part of the platform — images, video, audio, 3D models, textures, Blender renders — and manage those requests from initial queuing through GPU execution to final output storage and provenance recording. Importantly, Isis knows nothing about _why_ a given piece of content is being generated or how it will be used. It is a pure capability service: submit a job, get an output back, with a complete record of exactly how that output was produced. The system is organized as three cooperating REST API services (job submission, workflow management, and output tracking), a GPU worker process, a CLI, and a web front-end. A shared Redis queue connects the API to the workers, and Redis Streams carry lifecycle events to the rest of the platform. A single PostgreSQL database (via Prisma ORM) stores all persistent state. --- ## System Overview Isis provides: - **Job submission** via REST API (Hono, default port 3000) - **GPU worker processing** with ComfyUI, Blender, and custom executors - **Workflow management** with versioning and templates (port 3001) - **Output tracking** with full provenance and lineage graph (port 3002) - **Model registry** for checkpoint and LoRA management - **Client SDK** (`@isis/client`) and **CLI** for programmatic access Isis acts as the creative engine of the platform — other domains submit generation requests, and Isis manages the complete lifecycle from job queuing through GPU execution to output storage and provenance recording. ### High-Level Topology The following diagram shows how the five major runtime components relate to one another. The CLI and other domain clients talk to the three REST services; the generation API enqueues work to the GPU worker via Redis; and all services share a single PostgreSQL database and a MinIO/S3 object store. ``` +------------------+ | isis CLI | | (Commander) | +--------+---------+ | +----------------+----------------+ | | | +--------v--------+ +----v----------+ +----v-----------+ | generation-api | | workflow- | | output- | | Port 3000 | | registry | | registry | | (Hono) | | Port 3001 | | Port 3002 | | /api/v1/jobs | | (Hono) | | (Hono) | | /api/v1/models | | /api/v1/ | | /api/v1/ | +--------+--------+ | workflows | | outputs | | | /api/v1/ | | /api/v1/ | | | templates | | policies | | +---------------+ +----------------+ | +--------v--------+ | gpu-worker | | Redis queue | | GPU executors | | - texture | | - blender | | - gaussian | | - mesh | +-----------------+ | +--------v--------+ | PostgreSQL | (ISIS_DATABASE_URL) | (Prisma ORM) | +-----------------+ | +--------v--------+ | MinIO / S3 | (generated outputs) +-----------------+ | +--------v--------+ | Redis | (job queue + @oshun/event-bus) +-----------------+ ``` --- ## Service Topology ### Applications (6) Isis deploys as six distinct applications. Three are REST APIs that accept external requests; one is a background worker that processes GPU jobs; one is a CLI for scripted workflows; and one is a Next.js web front-end. REST services read their listen port from `process.env.PORT`, defaulting as shown below (`apps/isis/*/src/index.ts`). | Application | Type | Default port | Framework | Description | | ------------------- | ------------- | ------------ | ------------ | ----------------------------------------- | | `generation-api` | REST API | 3000 | Hono | Primary job submission and management API | | `workflow-registry` | REST API | 3001 | Hono | Workflow CRUD, versioning, templates | | `output-registry` | REST API | 3002 | Hono | Output manifests and retention policies | | `gpu-worker` | Worker | — | Node.js | GPU job processor with Redis poll | | `cli` | CLI Tool | — | Commander.js | Command-line interface for all services | | `web` | Web front-end | — | Next.js | `@isis/web` operator/creator front-end | --- ## Application Layer ### generation-api (default Port 3000) The generation API is the single entry point for all job submission. It authenticates requests, validates inputs with Zod, writes a `GenerationJob` record to the database, and enqueues the job for a GPU worker to pick up. It also exposes model management, workflow execution triggers, webhook subscriptions, and provider callbacks. **Middleware stack (Hono):** - CORS with configurable origins - Secure headers (CSP, HSTS, X-Frame-Options, etc.) - Pretty JSON for development - Rate limiting: 100 requests per 60 seconds on `/api/*` - JWT authentication middleware (fail-fast: requires `JWT_SECRET` in production) - Zod request/response validation via `@hono/zod-validator` **Core services:** - `JobService` — submit, list, cancel, get stats, queue health - `ModelService` — register, list, get, update, generate upload URLs - `ComfyUIService` — workflow execution via ComfyUI WebSocket protocol The app mounts five route groups: `jobs`, `models`, `workflows`, `webhook-subscriptions`, plus `/webhooks` (provider callbacks). The workflow routes validate a typed contract for each of 32 asset-pack workflow recipes. **Health endpoints:** - `GET /health` — Simple liveness probe (always 200 if process alive) - `GET /ready` — Readiness probe: runs job-service, model-service, and ComfyUI-service health checks in parallel and returns `200`/`503` with a per-dependency breakdown (jobs DB, models DB, queue, storage, provider) ### gpu-worker The GPU worker is a long-running Node.js process — it does not expose any HTTP port. Instead it polls a Redis queue continuously, deserializes incoming `JobEnvelope` messages, and routes each job to the appropriate executor. When a job finishes, the worker uploads the output to S3, writes `GeneratedOutput` and `Provenance` database records, and publishes a completion event on the event bus. The flow through the worker on every job is: ``` Poll Redis queue (ISIS_QUEUE_NAME) │ ▼ Deserialize JobEnvelope │ Route by GenerationType: ├── TEXTURE_UPSCALE → texture-upscale-executor ├── BLENDER_RENDER → blender-render-executor ├── GAUSSIAN_SPLATTING → gaussian-splatting-executor ├── MESH_PROCESSING → mesh-processing-executor └── All other types → generic executor (provider-routed) │ ▼ Execute (may call AI provider API) │ ▼ Upload output to S3 │ ▼ Create GeneratedOutput + Provenance records │ ▼ Update GenerationJob status → COMPLETED | FAILED │ ▼ Publish isis.job.completed event ``` **Configuration:** - `GPU_DEVICES` — comma-separated CUDA device indices - `WORKER_TYPE` — one of seven worker types (`texture-upscale`, `mesh-processing`, `blender-render`, `gaussian-splatting`, `video-generation`, `neural-bake`, `general`) - `QUEUE_NAME` — Redis queue to poll - `WORKER_ID` — unique worker identifier registered in `GpuWorker` table **Lifecycle events emitted** (internal): `worker:started`, `worker:stopped`, `job:started`, `job:progress`, `job:completed`, `job:failed` **Graceful shutdown:** On `SIGINT`/`SIGTERM`, the worker finishes the current job before exiting. ### workflow-registry (Port 3001) The workflow registry stores and versions ComfyUI workflow definitions. It is separate from the generation API so that workflow authoring tools and administrative interfaces can talk directly to it without going through the job submission path. It has two storage backends — `postgres` (production) and `in-memory` (test/dev override) — and exposes full-text search via PostgreSQL `tsvector` when database-backed. Key capabilities: - Full CRUD with semver versioning, changelog tracking, and star/favorite system - Visibility controls: `PRIVATE`, `TEAM`, `ORGANIZATION`, `PUBLIC` - Staging-recipe routes for promotion workflows ### output-registry (Port 3002) The output registry is the long-term home for every file that Isis generates. Rather than baking lifecycle management into the generation API, it is separated here so that storage policy decisions — tiering, retention, expiry — can evolve independently of job submission. - Tracks every output file with hash, dimensions, storage tier, and access statistics - **Storage tiering** — Automatic lifecycle management: HOT → WARM → COLD → GLACIER based on `RetentionPolicy` rules and access patterns - **Lineage graph** — Directed graph of `LineageEdge` records tracking derivation between outputs (e.g., upscaled-from, converted-from) - **Provenance** — Immutable `Provenance` records capturing full generation context for reproducibility - **File deduplication** — Content hash uniqueness prevents duplicate storage ### cli Commander.js CLI (`isis`) for scripted generation workflows and CI/CD integration. Authenticates via JWT token or API key stored in `~/.isis/config.yaml`. The CLI covers seven command groups: `generate`, `jobs`, `workflows`, `outputs`, `config`, `health`, and `auth`. --- ## Library Architecture The 55 libraries in `libs/isis/` are organized by concern. Understanding which group a library belongs to tells you what kind of dependency it is: core platform libraries are stable and depended on widely; provider adapters change when AI vendor APIs change; 3D generation libraries form a largely self-contained sub-system. What follows groups the 55 directories (54 TypeScript `@isis/*` packages plus one Python package) by that concern. ### Core platform (6) These six libraries form the foundation that every other Isis library or application may depend on. | Library | Purpose | | ----------------------- | ------------------------------------------------------------------------------------------------- | | `@isis/client` | TypeScript SDK: `IsisClient`, `HttpTransport`, `RequestBuilder`, typed errors | | `@isis/database` | Prisma ORM schema (1141 lines, 24 models, 14 enums) and generated client | | `@isis/job-envelope` | Job-envelope Zod schema, generation-type manifest, control vocabulary, composer dispatch payloads | | `@isis/workflows` | Workflow-registry domain library: definition storage, versioning, templates | | `@isis/outputs` | Output manifest management: file tracking, storage tiering, retention enforcement | | `@isis/event-publisher` | Typed event publishing for generation lifecycle events to `@oshun/event-bus` | ### AI providers, LLM, and orchestration (10) These libraries abstract every external AI vendor behind a common interface, and provide higher-level orchestration (batching, consensus, ReAct agents) on top. | Library | Purpose | | ------------------------------ | ---------------------------------------------------------------------------------------------------------- | | `@isis/ai-providers` | Unified provider adapters (LLM, image, video, video-processing, TTS, 3D, ComfyUI, Civitai, model-registry) | | `@isis/llm-providers` | Additional LLM provider adapters | | `@isis/llm-orchestrator` | LLM orchestration layer for multi-step workflows | | `@isis/batch-llm-processing` | Batch LLM inference for high-volume text tasks | | `@isis/token-budget` | LLM token-budget allocation, prediction, and enforcement | | `@isis/agent-consensus` | Multi-agent debate, reasoning, and consensus | | `@isis/react-framework` | ReAct-pattern (Reasoning + Acting) tool-using agent execution framework | | `@isis/prompt-engineering` | Prompt templates, builder, analyzer, optimizer | | `@isis/operation-orchestrator` | Multi-step operation orchestration: retry, dead-letter, pipeline/chain/phase model | | `@isis/managed-models` | Managed-model browser surface | ### ComfyUI and workflow authoring (5) ComfyUI is the primary visual workflow engine. These libraries cover everything from low-level WebSocket communication with a running ComfyUI instance up to higher-level workflow-class catalogs and custom Python nodes. | Library | Purpose | | ------------------------ | ----------------------------------------------------------------------------------------------------------------- | | `@isis/comfyui-sdk` | WebSocket-based ComfyUI SDK: server connection, workflow execution | | `@isis/comfyui-factory` | Workflow-class authoring, template diff, portability check, rehearsal harness | | `comfyui-nodes` | Python ComfyUI custom-node package (sacred-geometry, spiritual-styles, vfx-post, consciousness, lilith node sets) | | `@isis/3d-comfyui-nodes` | 3D-specific ComfyUI node definitions | | `@isis/workflow-classes` | Living-scene workflow-class catalogs (tara, nyx, veritas, …) | ### Image, video, and audio (11) This group covers the full media-generation surface area beyond simple text-to-image: AI video generation and enhancement, geometry extraction from video, all audio and voice synthesis capabilities, and AI texturing for 3D assets. | Library | Purpose | | -------------------------- | ------------------------------------------------------------------ | | `@isis/ai-video` | Video generation (providers, advanced, self-hosted) | | `@isis/video-enhancement` | AI video enhancement/restoration (super-resolution, interpolation) | | `@isis/video-to-mesh` | 3D-geometry extraction from video footage | | `@isis/audio-generation` | Audio, TTS, and voice generation with QA | | `@isis/music-generation` | Music generation: provider, guardrails, stems, provenance | | `@isis/voice-cloning` | Voice-cloning and synthesis orchestration | | `@isis/visual-dubbing` | Visual dubbing and dialogue-editing orchestration | | `@isis/face-synthesis` | Face synthesis, de-aging, and identity-transfer orchestration | | `@isis/seedance-provider` | Seedance video-generation provider integration | | `@isis/post-production-ai` | Per-production post-production intelligence for cinematic dailies | | `@isis/ai-texturing` | AI texture upscaling, seamless tiling, material-ID extraction | ### 3D generation (16) The 3D generation sub-system is the most complex part of Isis, covering multi-provider cloud generation, quality gating, rigging, post-processing, scene assembly, and governance. These 16 libraries form a largely self-contained 3D pipeline. | Library | Purpose | | --------------------------------- | -------------------------------------------------------------------------------------- | | `@isis/3d-generation` | Multi-provider 3D generation (Rodin, Meshy, Tripo, Trellis, Hunyuan, ThreeDFY, Marble) | | `@isis/3d-generation-benchmarks` | Cross-provider 3D quality and throughput benchmarking | | `@isis/3d-inference-local` | Self-hosted 3D model serving | | `@isis/3d-post-pipeline` | Mesh optimization, UV unwrapping, LOD generation, texture baking | | `@isis/3d-quality-gates` | Topology, UV-coverage, rig, and LOD-chain validation | | `@isis/3d-semantic-editing` | Text-guided semantic editing of generated 3D models | | `@isis/3d-asset-library` | 3D-asset browsing and management | | `@isis/3d-browser` | In-browser 3D model preview and interaction | | `@isis/3d-scene-assembly` | 3D scene-assembly modules | | `@isis/3d-marketplace-ops` | 3D-marketplace operations modules | | `@isis/3d-product-parity` | 3D product-parity modules | | `@isis/three-d-pipelines` | 3D pipeline-class, provider, topology, and provenance contracts | | `@isis/universal-rigging` | Automated skeletal rigging, skin weights, animation retargeting | | `@isis/gaussian-splatting` | NeRF/Gaussian-splatting reconstruction, mesh extraction | | `@isis/scene-from-image-composer` | End-to-end scene composition from a single image | | `@isis/model-governance-3d` | 3D model license tracking and compliance | ### Models, quality, and operations (7) The final group covers everything that happens around the edges of generation: fine-tuning models to produce new capabilities, detecting misuse, gating access by entitlement, surfacing outputs in a gallery, and managing the RunPod serverless GPU endpoint. | Library | Purpose | | ----------------------------- | ------------------------------------------------------------------------ | | `@isis/model-fine-tuning` | LoRA training and fine-tuning utilities | | `@isis/lora-training-surface` | LoRA training-run, model-merging, quality-view, lineage-tree surface | | `@isis/anomaly-detection` | Suspicious-activity detection, chargeback prediction, account protection | | `@isis/curated-cards` | Curated-card types, validators, preflight, entitlement gate | | `@isis/entitlements` | Generation-tier and studio-boundary entitlements | | `@isis/output-gallery` | Output-record gallery: lineage, branch-replay, bulk actions | | `@isis/runpod-surface` | RunPod endpoint registry, cost-quota, queue inspector, secret rotation | (The `apps/isis/web` front-end is packaged as `@isis/web`.) --- ## Data Flow Understanding how data moves through Isis is key to debugging and extending it. The three flows below cover the common case (single generation job), the workflow-driven case, and the output lineage tracking that links derived assets back to their origins. ### Generation Pipeline This is the most common path: a client submits a job, the API queues it, a GPU worker executes it, and the resulting file lands in object storage with its provenance recorded. ``` Client / CLI / Other Domain Isis Services Storage ───────────────────────── ────────────────────── ───────────────── POST /api/v1/jobs generation-api { │ type: TEXT_TO_IMAGE, │ validate JWT prompt: "...", │ validate Zod schema model: "sdxl-1.0" │ create GenerationJob (PENDING) } │ enqueue to Redis │ ← 202 Accepted │ { id: "job_xxx", status: PENDING }│ gpu-worker │ poll queue │ ├─ resolve ModelRegistry │ by modelId │ ├─ execute via │ ComfyUI / Executor │ (or AI provider API) │ ├─ emit job:progress events │ (progress 0→100%) │ ├─ upload output to S3 ──────────→ MinIO │ ├─ create GeneratedOutput │ {hash, dimensions, tier: HOT} │ ├─ create Provenance │ {model, params, seed, gpu, cost} │ ├─ update job → COMPLETED │ └─ publish isis.asset.generated ──→ event bus output-registry │ store manifest │ evaluate retention policy │ queue tier transitions ``` ### Workflow Execution Flow When a job is submitted with a `workflowId`, the generation API loads the workflow definition from the workflow registry, validates the job parameters against the workflow's schema, and enqueues the job with the full workflow context attached. The GPU worker then drives ComfyUI over WebSocket to execute the graph. ``` User workflow-registry generation-api ──── ───────────────── ────────────── Create workflow with nodes/edges ──→ Store definition + version (1.0.0) │ Submit job with │ workflowId ─────────────────────────────────→ Load workflow definition from DB │ Validate params against schema │ Enqueue with workflow context │ gpu-worker │ Execute via ComfyUI WebSocket │ Store output + provenance ``` ### Output Lineage Flow Every derived output — an upscaled image, a converted mesh, an engine-native asset — is linked back to its origin via `LineageEdge` records. This makes it possible to trace any asset backwards to its original generation job and parameters, and to reconstruct the full derivation chain for compliance or reproducibility purposes. ``` Original Image Upscaled Engine Asset ───────────── ──────── ──────────── GeneratedOutput ─UPSCALED_FROM─→ GeneratedOutput (job: t2i) (job: upscale) (512×512) (2048×2048) │ CONVERTED_FROM │ ▼ GeneratedOutput (.uasset format) bellonaAssetId set in Yemaya Asset record ``` --- ## Cross-Domain Integration Isis is a pure capability domain — it provides generation services to other domains and does not depend on any other domain. This boundary is deliberate: if Isis depended on Yemaya or Bellona, a change to those domains could break generation for everyone. Instead, Isis speaks only in opaque IDs (`projectId`, `userId`) that originating domains provide and receive back. ### Event Bus Architecture Isis uses `@isis/event-publisher` (wrapping `@oshun/event-bus`) to publish lifecycle events on Redis Streams under the `oshun:events` key prefix. Any domain interested in generation outcomes subscribes to these events rather than polling the Isis API: ``` Isis generation-api / gpu-worker │ └── @isis/event-publisher │ └── @oshun/event-bus (Redis Streams, keyPrefix "oshun:events") │ ├── isis.job.* events → job lifecycle (queued/started/ │ progress/completed/failed/cancelled) ├── isis.asset.generated → new asset available ├── isis.workflow.* → workflow registered/updated └── isis.model.loaded → model loaded onto a GPU worker ``` Event types and payload schemas are defined canonically in `@oshun/contracts` (`IsisEventTypes`). Event publishing is best-effort: a publish failure is logged and swallowed so it never breaks the generation flow. ### Inbound Integration Other domains (Yemaya, Hathor, Bellona, Lilith) submit generation jobs directly to the Isis REST API or via the `@isis/client` SDK. Jobs carry a `projectId` cross-reference that is stored verbatim on the `GenerationJob` record. Isis never resolves what that project represents — that is always the caller's responsibility. The echoed `projectId` in completion events is how the originating domain correlates results back to its own records: ``` Yemaya publish yemaya.generation.requested → Isis handler creates GenerationJob{projectId: "prj_xyz"} → Job executes, output stored → Isis publishes isis.job.completed{projectId: "prj_xyz"} → Yemaya handler resolves Project by projectId, creates Asset ``` --- ## Canary Rollout and Operational Tooling Isis includes production-grade operational tooling for safe rollouts and incident management, documented in `docs/domains/isis/remediation/` and `docs/domains/isis/remediation-v2/`. ### Canary Queue Rollout Rather than doing a hard cutover when the queue producer changes, Isis supports a gradual canary rollout controlled by environment variables. Traffic is split between the old and new queue at a configured percentage, and SLO gates determine whether to promote or roll back: ```bash ISIS_QUEUE_PRODUCER_ENABLED=true # Master switch ISIS_QUEUE_PRODUCER_CANARY_ENABLED=false # Canary mode ISIS_QUEUE_PRODUCER_CANARY_PERCENTAGE=10 # 10% of traffic to new queue ISIS_QUEUE_PRODUCER_CANARY_PROJECT_IDS= # Allowlist for canary routing ``` Canary advancement is automated via `scripts/isis/advance_queue_canary_rollout.sh` with SLO-gated promotion/rollback decisions. ### Operational Scripts A set of shell scripts in `scripts/isis/` supports the full release and validation lifecycle: | Script | Purpose | | --------------------------------- | ----------------------------------------------------- | | `validate_compose_app_paths.sh` | Validate Docker Compose path wiring | | `smoke_compose_health.sh` | Run `/health` + `/ready` smoke checks | | `verify_canary_slo_cost.sh` | Verify SLO/cost release gate from job history | | `advance_queue_canary_rollout.sh` | Advance canary with promote/rollback decision | | `run_release_gameday.sh` | Pre-cutover game-day harness, emits structured report | | `run_post_cutover_validation.sh` | Post-cutover validation with sign-off evidence | ### SLO Definitions The following SLOs are defined in `docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.json` and gate canary promotion decisions: - Job submission success rate: >99.5% - Job completion latency (p99): type-dependent (image < 30s, 3D < 300s) - Output storage durability: >99.999% - API availability: >99.9% --- ## Deployment Architecture ### Container Strategy Each Isis service deploys as its own container image. The API services are stateless and scale horizontally behind a load balancer; the GPU worker scales by queue depth and is tied to GPU node pools. | Service | Image | Scaling | | ------------------- | --------------- | ------------------------------------ | | `generation-api` | `isis-api` | Horizontal, stateless, load balanced | | `gpu-worker` | `isis-worker` | GPU node pools, scale by queue depth | | `workflow-registry` | `isis-workflow` | Horizontal, stateless | | `output-registry` | `isis-output` | Horizontal, stateless | ### GPU Infrastructure GPU workers are deployed on GPU-enabled nodes. The `GPU_DEVICES` environment variable specifies which CUDA devices the worker can use. Multiple worker instances can run on the same node (one per GPU) or across multiple nodes. RunPod serverless is used for on-demand GPU capacity spikes without maintaining persistent GPU instances (per ADR-0003). ### Development Stack To bring up Isis locally with GPU support: ```bash # Start Isis local stack with GPU profile docker compose -f docker/docker-compose.yml --profile gpu up -d \ postgres redis qdrant isis-api isis-generator ``` --- ## Technology Stack The following table records the technology choice for each layer and the rationale behind it. Cross-referencing this with the ADR list (in the specifications doc) gives the full decision history. | Layer | Technology | Rationale | | ---------------- | ------------------------------------------------------- | --------------------------------------------------- | | API Framework | Hono | Lightweight TypeScript-native REST framework | | ORM | Prisma | Type-safe queries, migrations, 24-model schema | | Database | PostgreSQL | Relational data with JSONB for parameters | | Job Queue | Redis (poll model) | Reliable delivery; poll chosen over push (ADR-0001) | | Event Bus | Redis Streams (`@oshun/event-bus`) | Cross-domain typed event publishing | | Object Storage | MinIO (dev) / S3 (prod) | S3-compatible output storage | | AI Image | Stability AI, Black Forest Labs (FLUX) | Stable Diffusion / SD3.5 and FLUX models | | AI Video | Hunyuan, LTX, Wan, Seedance | Text-to-video and image-to-video providers | | AI 3D | Rodin, Meshy, Tripo, Trellis, Hunyuan, ThreeDFY, Marble | Multi-provider 3D generation | | AI Audio | ElevenLabs | High-quality voice synthesis | | AI Text | Anthropic, OpenAI, Google AI, xAI, Ollama | Claude, GPT, Gemini, Grok, and local LLM models | | Video Processing | NVIDIA RTX, RIFE, Topaz | Super-resolution, frame interpolation, enhancement | | Workflow Engine | ComfyUI | Visual workflow design with WebSocket execution | | Validation | Zod + `@hono/zod-validator` | Runtime validation + type inference | | Build | Nx (`@nx/js:tsc`) | Monorepo-integrated TypeScript build | | Testing | Vitest | Fast unit and integration testing | | Logging | Pino (`@oshun/logging`) | Structured JSON logging | --- ## Design Principles The following principles explain _why_ the architecture is shaped the way it is, not just _what_ it does. Each one reflects a deliberate trade-off made during the system's design. 1. **Generation-as-a-service** — Isis is a pure capability service with no domain knowledge about what the content will be used for. Job context (projectId, userId) flows through as opaque references. This isolation means changes to Yemaya or Bellona's data models never require changes to Isis. 2. **Full provenance for every output** — Every generated file has an immutable `Provenance` record capturing model, parameters, seed, and cost. This enables reproducibility, cost attribution, and compliance auditing. You can always answer "exactly how was this file produced?" 3. **Lineage over mutation** — Outputs are never modified in place. Transformations (upscaling, conversion) create new `GeneratedOutput` records linked by `LineageEdge` records. The lineage graph is append-only, which means the history of every asset is complete and unambiguous. 4. **Provider abstraction** — `@isis/ai-providers` abstracts all AI provider differences behind a uniform interface. Generation type determines which providers are eligible; the system routes to the best available provider. Adding a new provider requires only a new adapter, not changes to job submission logic. 5. **Poll-based queue for reliability** — The Redis poll model (ADR-0001) was chosen over pub/sub push because it provides better at-least-once delivery guarantees and makes worker restarts safe without job loss. 6. **Canary-first deployments** — All queue and API changes go through canary rollout with SLO-gated promotion. Rollback criteria are pre-defined and automated.