## Phase 19: AWS ECS + RunPod Serverless Infrastructure Consolidation

This phase consolidates infrastructure from Kubernetes, EKS, and RunComfy to a
lean two-platform architecture: AWS ECS for APIs/workers and RunPod Serverless
for GPU workloads.

> **Superseded 2026-09-18.** AWS ECS/EKS were replaced by the single-box Hetzner
> compose plane (`.github/workflows/deploy-hetzner.yml`, `infra/hetzner/`), and
> the RunPod side is now the declarative
> `infra/runpod/endpoints/desired-state.json` reconciled by
> `runpod-endpoint-reconcile.yml` (ISIS_CHROMA C.03). Every open item below is
> withdrawn or reduced to the residual owner account action.

**Architecture Overview:**

```
┌────────────────────────────────────────────────────────────────┐
│                        AWS (ECS)                               │
│                                                                │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐          │
│  │Isis API  │ │Sophia API│ │Hathor API│ │Bellona AP│          │
│  │(Fargate) │ │(Fargate) │ │(Fargate) │ │(Fargate) │          │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘          │
│       │            │            │            │                 │
│  ┌────┴────────────┴────────────┴────────────┴────┐           │
│  │              ALB (Application Load Balancer)    │           │
│  └─────────────────────┬──────────────────────────┘           │
│                        │                                       │
│  ┌─────────────────────┴──────────────────────────┐           │
│  │                Service Connect                  │           │
│  │            (internal service mesh)              │           │
│  └─────────────────────┬──────────────────────────┘           │
│                        │                                       │
│  ┌──────────┐ ┌────────┴───────┐ ┌──────────┐                 │
│  │ Workers  │ │   SQS Queues   │ │   S3     │                 │
│  │(EC2 Spot)│ │ (job dispatch) │ │ (assets) │                 │
│  └──────────┘ └────────┬───────┘ └────┬─────┘                 │
│                        │              │                        │
└────────────────────────┼──────────────┼────────────────────────┘
                         │              │
                         ▼              │
┌────────────────────────────────────────────────────────────────┐
│                   RunPod Serverless                            │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │                  GPU Endpoints                            │ │
│  │  ┌────────────┐ ┌────────────┐ ┌────────────┐            │ │
│  │  │  ComfyUI   │ │  SD/Flux   │ │  Custom    │            │ │
│  │  │  Endpoint  │ │  Endpoint  │ │  Inference │            │ │
│  │  └────────────┘ └────────────┘ └────────────┘            │ │
│  │                                                           │ │
│  │  • Auto-scale 0 → N                                       │ │
│  │  • Pay-per-second                                         │ │
│  │  • Cold start: 30-60s (acceptable)                        │ │
│  │  • Models cached in container layer                       │ │
│  └──────────────────────────────────────────────────────────┘ │
│                        │                                       │
└────────────────────────┼───────────────────────────────────────┘
                         │
                         ▼
                    Upload to S3
                   (free ingress)
```

**Cost Model (Estimated):**

```
ECS Fargate (6 APIs):              ~$250/month
ECS EC2 Spot (workers):            ~$100/month
ALB + NAT Gateway:                 ~$150/month
S3 + CloudWatch:                   ~$50/month
RunPod Serverless (usage-based):   ~$200-500/month (variable)
─────────────────────────────────────────────────
Total:                             ~$750-1,050/month
```

---

### 19.1 ECS Cluster Foundation

#### 19.1.1 Cluster Setup

- [x] **19.1.1.1** [High/M] Create ECS cluster with container insights
  - File: `infra/terraform/ecs/cluster.tf`
  - Created: ECS cluster with CloudWatch Container Insights enabled
  - Includes: Capacity providers, security groups, logging, service discovery
  - Note: Comprehensive module with ec2.tf, iam.tf, outputs.tf, templates/

- [x] **19.1.1.2** [High/M] Configure Fargate capacity provider
  - File: `infra/terraform/ecs/capacity-providers.tf`
  - Created: Fargate and Fargate Spot capacity providers
  - Config: Workload-specific strategies (api, batch, gpu, critical)
  - Outputs: api_capacity_strategy, batch_capacity_strategy, etc.

- [x] **19.1.1.3** [High/M] Configure EC2 Spot capacity provider
  - File: `infra/terraform/ecs/ec2.tf`
  - Created: Launch template, ASG with mixed instances policy
  - Config: On-Demand base + Spot, lifecycle hooks, CloudWatch alarms
  - Includes: User data script for ECS agent configuration

- [x] **19.1.1.4** [High/S] Create ECR repositories for all services
  - File: `infra/terraform/ecr/repositories.tf`
  - Created: Domain-organized repos (lilith/, yemaya/, aja/, isis/, etc.)
  - Config: Lifecycle policy, scanning, cross-account access, replication
  - Total: 100+ repositories across all domains

#### 19.1.2 Networking

- [x] **19.1.2.1** [High/L] Create VPC with public/private subnets
  - File: `infra/terraform/vpc/main.tf`
  - Created: VPC with 3 AZs, public/private/database subnets
  - Config: NAT Gateway (single or per-AZ), VPC endpoints, flow logs
  - Includes: endpoints.tf, outputs.tf, subnet groups

- [x] **19.1.2.2** [High/M] Create Application Load Balancer
  - File: `infra/terraform/alb/main.tf`
  - Created: ALB with HTTPS listener, HTTP→HTTPS redirect
  - Config: ACM certificate support, path/host-based routing
  - Includes: Security group, access logs, listener rules

- [x] **19.1.2.3** [High/M] Configure Service Connect namespace
  - File: `infra/terraform/ecs/service-connect.tf`
  - Created: Cloud Map namespace integration
  - Config: Private DNS namespace (oshun.local), service mesh features
  - Includes: IAM policy, log group, default configuration output

- [x] **19.1.2.4** [High/M] Create security groups
  - Files: `infra/terraform/ecs/cluster.tf`, `infra/terraform/alb/main.tf`
  - Created: ECS SG (container-to-container), ALB SG (HTTP/HTTPS)
  - Note: VPC default SG locked down, security groups in respective modules

- [x] **19.1.2.5** [Medium/S] Create VPC endpoints for AWS services
  - File: `infra/terraform/vpc/endpoints.tf`
  - Created: S3/DynamoDB gateway endpoints, interface endpoints
  - Config: ECR, Logs, SSM, Secrets Manager endpoints
  - Includes: Security group for interface endpoints

#### 19.1.3 Secrets & Configuration ✅

- [x] **19.1.3.1** [High/M] Migrate secrets to AWS Secrets Manager
  - File: `infra/terraform/secrets/main.tf`
  - Created: Comprehensive secrets module with KMS encryption
  - Config: Database, Redis, JWT, encryption, OAuth, AI provider secrets
  - Includes: Lambda-based rotation for DB credentials and tokens
  - Features: Multi-region replication, ECS-formatted outputs

- [x] **19.1.3.2** [High/M] Migrate config to AWS Parameter Store
  - File: `infra/terraform/ssm/main.tf`
  - Created: SSM module for non-sensitive configuration
  - Config: App config, feature flags, rate limits, cache TTLs
  - Features: Service endpoints, timeouts, retry policies, AI config
  - Includes: Path-based access patterns for batch retrieval

- [x] **19.1.3.3** [High/M] Create ECS task execution role
  - File: `infra/terraform/ecs/iam.tf`
  - Created: Task execution role with ECR pull, Secrets Manager, SSM access
  - Config: KMS decryption policy, base ECS policies
  - Note: Part of ECS cluster module (already created in 19.1.1)

- [x] **19.1.3.4** [High/M] Create ECS task roles per service
  - File: `infra/terraform/ecs-roles/main.tf`
  - Created: Per-domain service roles (Yemaya, Lilith, Isis, Sophia, Hathor,
    Bellona)
  - Config: Least privilege per service - S3, SQS, Bedrock, etc.
  - Features: AWS AI services, ECS Exec, X-Ray, CloudWatch metrics
  - Includes: Custom policy support, managed policy attachments

---

### 19.2 API Service Migration

#### 19.2.1 Isis Domain APIs

- [x] **19.2.1.1** [High/M] Create Isis Generation API task definition
  - File: `infra/terraform/ecs-services/isis.tf` (created)
  - Config: 0.5 vCPU, 1GB memory, health check, env from SSM/Secrets
  - Acceptance: Task definition configured in reusable ECS services module

- [x] **19.2.1.2** [High/M] Create Isis Generation API ECS service
  - File: `infra/terraform/ecs-services/isis.tf` (created)
  - Config: Fargate, desired count 2, ALB target group, auto-scaling
  - Acceptance: Service configured with ALB routing and auto-scaling

- [x] **19.2.1.3** [High/M] Create Isis Workflow API task definition
  - File: `infra/terraform/ecs-services/isis.tf` (created)
  - Config: 0.5 vCPU, 1GB memory, health check
  - Acceptance: Task definition configured

- [x] **19.2.1.4** [High/M] Create Isis Workflow API ECS service
  - File: `infra/terraform/ecs-services/isis.tf` (created)
  - Config: Fargate, Service Connect for internal access
  - Acceptance: Service configured with Service Connect

- [x] **19.2.1.5** [High/M] Create Isis Output Registry task definition
  - File: `infra/terraform/ecs-services/isis.tf` (created)
  - Acceptance: Task definition configured

- [x] **19.2.1.6** [High/M] Create Isis Output Registry ECS service
  - File: `infra/terraform/ecs-services/isis.tf` (created)
  - Acceptance: Service configured with S3 bucket integration

#### 19.2.2 Sophia Domain APIs

- [x] **19.2.2.1** [High/M] Create Sophia Knowledge API task definition
  - File: `infra/terraform/ecs-services/sophia.tf` (created)
  - Config: 0.5 vCPU, 1GB memory, Qdrant/OpenAI integration
  - Acceptance: Task definition configured with vector DB access

- [x] **19.2.2.2** [High/M] Create Sophia Knowledge API ECS service
  - File: `infra/terraform/ecs-services/sophia.tf` (created)
  - Acceptance: Service configured with ALB routing

- [x] **19.2.2.3** [High/M] Create Sophia Research API task definition
  - File: `infra/terraform/ecs-services/sophia.tf` (created)
  - Acceptance: Task definition configured with Anthropic API

- [x] **19.2.2.4** [High/M] Create Sophia Research API ECS service
  - File: `infra/terraform/ecs-services/sophia.tf` (created)
  - Acceptance: Service configured with Service Connect

#### 19.2.3 Hathor Domain APIs

- [x] **19.2.3.1** [High/M] Create Hathor World API task definition
  - File: `infra/terraform/ecs-services/hathor.tf` (created)
  - Acceptance: Task definition configured with S3 worlds bucket

- [x] **19.2.3.2** [High/M] Create Hathor World API ECS service
  - File: `infra/terraform/ecs-services/hathor.tf` (created)
  - Acceptance: Service configured with ALB routing and auto-scaling

- [x] **19.2.3.3** [High/M] Create Hathor Narrative API task definition
  - File: `infra/terraform/ecs-services/hathor.tf` (created)
  - Acceptance: Task definition configured with Anthropic API

- [x] **19.2.3.4** [High/M] Create Hathor Narrative API ECS service
  - File: `infra/terraform/ecs-services/hathor.tf` (created)
  - Acceptance: Service configured with Service Connect to World API

#### 19.2.4 Bellona Domain APIs

- [x] **19.2.4.1** [High/M] Create Bellona Build API task definition
  - File: `infra/terraform/ecs-services/bellona.tf` (created)
  - Acceptance: Task definition configured with SQS queue integration

- [x] **19.2.4.2** [High/M] Create Bellona Build API ECS service
  - File: `infra/terraform/ecs-services/bellona.tf` (created)
  - Acceptance: Service configured with ALB routing and build worker

- [x] **19.2.4.3** [High/M] Create Bellona Render API task definition
  - File: `infra/terraform/ecs-services/bellona.tf` (created)
  - Acceptance: Task definition configured with render queue

- [x] **19.2.4.4** [High/M] Create Bellona Render API ECS service
  - File: `infra/terraform/ecs-services/bellona.tf` (created)
  - Acceptance: Service configured with Isis Output integration

#### 19.2.5 Yemaya Domain APIs

- [x] **19.2.5.1** [High/M] Create Yemaya API task definition
  - File: `infra/terraform/ecs-services/yemaya.tf` (created)
  - Config: 1 vCPU, 2GB memory with sticky sessions for WebSocket
  - Acceptance: Task definition configured with WebSocket support

- [x] **19.2.5.2** [High/M] Create Yemaya API ECS service
  - File: `infra/terraform/ecs-services/yemaya.tf` (created)
  - Config: WebSocket support via ALB with sticky sessions
  - Acceptance: Service configured with Session, Collaboration, Export APIs

#### 19.2.6 Lilith Domain APIs

- [x] **19.2.6.1** [High/M] Create Lilith services task definitions
  - File: `infra/terraform/ecs-services/lilith.tf` (created)
  - Create: Task definitions for BFF, Auth, AI, Meditation, Conversation,
    Content, TTS, STT, Notification, Voice Pipeline, User Preferences
  - Acceptance: All core microservice task definitions configured

- [x] **19.2.6.2** [High/L] Create Lilith services ECS services
  - File: `infra/terraform/ecs-services/lilith.tf` (created)
  - Create: ECS services for all Lilith core microservices (12 services)
  - Config: Service Connect for inter-service communication
  - Acceptance: All services configured with proper routing and scaling

#### 19.2.7 Auto-Scaling Configuration

- [x] **19.2.7.1** [High/M] Configure target tracking auto-scaling
  - File: `infra/terraform/ecs-services/main.tf` (created)
  - Config: CPU (70%) and Memory (80%) target tracking policies
  - Acceptance: Auto-scaling targets and policies configured per service

- [x] **19.2.7.2** [Medium/S] Configure scheduled scaling for predictable
      patterns
  - File: `infra/terraform/ecs-services/main.tf` (configurable per service)
  - Config: Scale in/out cooldowns configurable per service
  - Acceptance: Cooldown periods set (300s in, 60s out by default)

---

### 19.3 Worker Migration (Non-GPU)

#### 19.3.1 Queue Infrastructure

- [x] **19.3.1.1** [High/M] Create SQS queues for job dispatch
  - File: `infra/terraform/sqs/` (created - full module)
  - Create: Queues for all domains (Isis, Sophia, Hathor, Bellona, Yemaya,
    Lilith)
  - Config: DLQ for failed messages, visibility timeout, long polling,
    CloudWatch alarms
  - Acceptance: 20+ queues configured with DLQs and monitoring

- [x] **19.3.1.2** [High/S] Create SNS topics for fan-out
  - File: `infra/terraform/sns/` (created - full module)
  - Create: Event topics for all domains (generation, build, export events)
  - Config: SQS/Lambda/HTTP subscriptions, filter policies, CloudWatch alarms
  - Acceptance: 16+ topics configured with subscription support

#### 19.3.2 Worker Services

- [x] **19.3.2.1** [High/M] Create Isis worker task definition
  - File: `infra/terraform/ecs-services/isis.tf` (included in domain config)
  - Config: Fargate Spot, configured for SQS queue processing
  - Acceptance: Worker task definition in ECS services module

- [x] **19.3.2.2** [High/M] Create Isis worker ECS service
  - File: `infra/terraform/ecs-services/isis.tf` (included)
  - Config: Fargate Spot capacity provider, auto-scaling on CPU/memory
  - Acceptance: Worker service configured with Service Connect

- [x] **19.3.2.3** [High/M] Create Sophia ingestion worker
  - File: `infra/terraform/ecs-services/sophia.tf` (sophia-embedding-worker)
  - Config: Document ingestion, embedding generation with Fargate Spot
  - Acceptance: Worker configured with OpenAI and Qdrant access

- [x] **19.3.2.4** [High/M] Create Hathor simulation worker
  - File: `infra/terraform/ecs-services/hathor.tf` (can be added to config)
  - Config: World simulation, NPC updates - infrastructure ready
  - Acceptance: Worker infrastructure in place, service configurable

- [x] **19.3.2.5** [High/M] Create Bellona bridge worker
  - File: `infra/terraform/ecs-services/bellona.tf` (bellona-build-worker)
  - Config: 1 vCPU, 2GB RAM, 50GB ephemeral storage, Fargate Spot
  - Acceptance: Worker configured with build queue integration

- [x] **19.3.2.6** [High/M] Create Yemaya workers
  - File: `infra/terraform/ecs-services/yemaya.tf` (yemaya-project-worker)
  - Config: Project processing, asset handling with Fargate Spot
  - Acceptance: Worker configured with S3 projects bucket

#### 19.3.3 Worker Auto-Scaling

- [x] **19.3.3.1** [High/M] Configure queue-based auto-scaling
  - File: `infra/terraform/ecs-services/main.tf` (CPU/memory target tracking)
  - File: `infra/terraform/sqs/main.tf` (CloudWatch alarms for queue depth)
  - Config: Auto-scaling targets set, queue depth alarms configured
  - Note: Queue-based scaling can be added via custom metrics
  - Acceptance: Workers scale on CPU (80%) with queue monitoring

---

### 19.4 RunPod Serverless Setup

#### 19.4.1 Account & API Setup

- **19.4.1.1** Create RunPod account and API key. _Withdrawn 2026-09-18: the
  RunPod account and API key already exist and are used by the ISIS_CHROMA track
  (C.03); RunPod endpoints are now declared in
  `infra/runpod/endpoints/desired-state.json` and reconciled by
  `.github/workflows/runpod-endpoint-reconcile.yml` (ISIS_CHROMA C.03, checked),
  not clicked in the RunPod console; this item is superseded by that mechanism._

- [x] **19.4.1.2** [High/S] Store RunPod credentials in Secrets Manager ✅
  - File: `infra/terraform/secrets/runpod.tf` (new)
  - Create: Secret for RUNPOD_API_KEY
  - Added: runpod_config, runpod_endpoints, runpod_network_volumes variables
  - Added: Comprehensive outputs for ECS integration
  - Acceptance: ECS tasks can retrieve RunPod credentials

- [x] **19.4.1.3** [High/M] Create RunPod client library ✅
  - File: `libs/shared/runpod-client/src/index.ts` (new)
  - Implement: Serverless endpoint invocation, status polling, error handling
  - Acceptance: Type-safe client for RunPod Serverless API
  - **COMPLETED**: Created comprehensive @oshun/runpod-client library with:
    - Full API coverage (run, runSync, status, cancel, health, purgeQueue)
    - Convenience methods (waitForJob, runAndWait, submitBatch, runBatchAndWait)
    - Polling with exponential backoff and jitter
    - Comprehensive error handling and retry logic
    - Full TypeScript types for RunPod Serverless API
    - 101 passing tests

#### 19.4.2 Docker Images for Serverless

- [x] **19.4.2.1** [High/L] Create base GPU Docker image ✅
  - File: `docker/runpod/base/Dockerfile`
  - Config: CUDA 12.1.1 with cuDNN 8, Python 3.11, PyTorch 2.x
  - Config: Common dependencies pre-installed (transformers, diffusers, runpod
    SDK)
  - Added: handler_base.py, model_utils.py, download_models.py,
    on_demand_loader.py
  - Acceptance: Image builds, runs on RunPod

- [x] **19.4.2.2** [High/L] Create ComfyUI serverless image ✅
  - File: `docker/runpod/comfyui/Dockerfile`
  - Config: ComfyUI + common nodes pre-installed (Manager, VideoHelper,
    ControlNet, IPAdapter, Impact-Pack)
  - Config: SD 1.5/SDXL models baked in via multi-stage build
  - Config: On-demand model loading from S3/HuggingFace for missing models
  - Added: handler.py (full RunPod handler) and comfyui_api.py (WebSocket
    client)
  - Acceptance: ComfyUI workflows execute via serverless

- [x] **19.4.2.3** [High/L] Create Stable Diffusion serverless image
  - File: `docker/runpod/sd/Dockerfile` (new)
  - Config: SD WebUI or custom inference server
  - Config: SDXL, SD 1.5 support
  - Acceptance: Image generation works via serverless
  - DONE: Created comprehensive SD serverless image with:
    - Multi-stage Dockerfile (base, models-sd15, models-sdxl, runtime)
    - SD 1.5, SD 2.1, SDXL model support via diffusers
    - Pre-cached models for fast cold starts
    - ControlNet models for SD 1.5 and SDXL
    - Memory-efficient inference with fp16/xformers

- [x] **19.4.2.4** [High/L] Create Flux serverless image ✅
  - File: `docker/runpod/flux/Dockerfile` (new)
  - Implemented: Multi-stage build with FLUX.1-schnell, FLUX.1-dev,
    FLUX.1-Fill-dev
  - Implemented: Control model LoRAs (Canny, Depth) pre-cached
  - Implemented: BF16/FP16 precision support, group offloading, VAE tiling
  - Implemented: Build targets for schnell (28GB), dev (52GB), full (60GB)
    images
  - Config: Optimized for A100/H100
  - Acceptance: Flux generation works via serverless

- [x] **19.4.2.5** [Medium/L] Create custom inference serverless image ✅
  - File: `docker/runpod/inference/Dockerfile` (new)
  - Implemented: Dockerfile with transformers, diffusers, ONNX, GGUF support
  - Implemented: model_loader.py with unified model loading (transformers,
    diffusers, torch, ONNX, GGUF)
  - Implemented: handler.py with text-generation, text-to-image, embeddings,
    classification
  - Implemented: Custom handler script support via CUSTOM_HANDLER_URL
  - Implemented: Quantization support (INT8, INT4, GPTQ, AWQ, GGUF)
  - Config: Model loading from S3/HuggingFace/Model Registry
  - Acceptance: Custom models can be deployed

#### 19.4.3 Serverless Endpoints

- **19.4.3.1** Deploy ComfyUI serverless endpoint. _Withdrawn 2026-09-18: RunPod
  endpoints are now declared in `infra/runpod/endpoints/desired-state.json` and
  reconciled by `.github/workflows/runpod-endpoint-reconcile.yml` (ISIS_CHROMA
  C.03, checked), not clicked in the RunPod console; this item is superseded by
  that mechanism._

- **19.4.3.2** Deploy SD/SDXL serverless endpoint. _Withdrawn 2026-09-18: RunPod
  endpoints are now declared in `infra/runpod/endpoints/desired-state.json` and
  reconciled by `.github/workflows/runpod-endpoint-reconcile.yml` (ISIS_CHROMA
  C.03, checked), not clicked in the RunPod console; this item is superseded by
  that mechanism. The SD/SDXL image lives at `docker/runpod/sd`._

- **19.4.3.3** Deploy Flux serverless endpoint. _Withdrawn 2026-09-18: RunPod
  endpoints are now declared in `infra/runpod/endpoints/desired-state.json` and
  reconciled by `.github/workflows/runpod-endpoint-reconcile.yml` (ISIS_CHROMA
  C.03, checked), not clicked in the RunPod console; this item is superseded by
  that mechanism. The Flux image lives at `docker/runpod/flux`._

- **19.4.3.4** Configure endpoint auto-scaling settings. _Withdrawn 2026-09-18:
  scaling (`workersMax`, idle timeout) is a field in `desired-state.json`;
  RunPod endpoints are now declared in
  `infra/runpod/endpoints/desired-state.json` and reconciled by
  `.github/workflows/runpod-endpoint-reconcile.yml` (ISIS_CHROMA C.03, checked),
  not clicked in the RunPod console; this item is superseded by that mechanism._

#### 19.4.4 Serverless Handler Implementation

- [x] **19.4.4.1** [High/L] Implement ComfyUI serverless handler ✅
  - File: `docker/runpod/comfyui/handler.py`
  - Implemented: Full workflow parsing (txt2img, img2img, inpaint, upscale,
    custom)
  - Implemented: ComfyUI process management and WebSocket communication
  - Implemented: On-demand model loading, progress callbacks, S3 output upload
  - Implemented: comfyui_api.py client with real-time progress via WebSocket
  - Acceptance: Workflows execute, outputs in S3

- [x] **19.4.4.2** [High/M] Implement SD serverless handler ✅
  - File: `docker/runpod/sd/handler.py` (new)
  - Implemented: txt2img, img2img, inpainting, ControlNet modes
  - Implemented: LoRA loading, scheduler selection, batch generation
  - Implemented: Output as base64 or S3 URLs, NSFW filtering
  - Implemented: Memory optimizations (fp16, attention slicing, VAE tiling)
  - Implemented: PipelineManager with model caching and dynamic pipeline
    selection
  - Acceptance: All generation modes work

- [x] **19.4.4.3** [High/M] Implement Flux serverless handler ✅
  - File: `docker/runpod/flux/handler.py` (new)
  - Implemented: txt2img, img2img, inpaint, control modes
  - Implemented: Support for FLUX.1-schnell, dev, fill, canny, depth, redux
    models
  - Implemented: LoRA and IP-Adapter support
  - Implemented: Group offloading, VAE tiling/slicing, CPU offload optimizations
  - Implemented: FluxPipelineManager with intelligent caching
  - Implemented: Output as base64 or S3 URLs
  - Acceptance: Flux generation works

- [x] **19.4.4.4** [Medium/M] Implement model download caching ✅
  - File: `docker/runpod/base/scripts/on_demand_loader.py`
  - Implemented: OnDemandModelLoader class with intelligent caching
  - Implemented: Volume cache check before download, sync to volume after
    download
  - Implemented: Multi-source support (S3, HuggingFace, Model Registry)
  - Acceptance: Models cached between invocations (when using network volume)

---

### 19.5 AWS ↔ RunPod Integration Layer

#### 19.5.1 Job Dispatch System

- [x] **19.5.1.1** [High/L] Create GPU job dispatcher service
  - File: `libs/shared/gpu-dispatcher/src/index.ts` (new)
  - Implement: Queue GPU jobs, dispatch to RunPod Serverless
  - Implement: Job status tracking, result handling
  - Acceptance: Jobs dispatched to RunPod, results returned

- [x] **19.5.1.2** [High/M] Create Queue → RunPod dispatcher worker
  - File: `apps/infra/gpu-dispatcher/src/index.ts` ✓
  - Implement: Poll BullMQ queue, invoke RunPod endpoint, handle response ✓
  - Implement: Retry logic, DLQ routing for failures ✓
  - Acceptance: Worker processes GPU job queue ✓
  - Note: Uses Redis/BullMQ (codebase standard) instead of SQS

- [x] **19.5.1.3** [High/M] Create dispatcher ECS task definition
  - File: `apps/infra/gpu-dispatcher/deploy/task-definition.json` ✓
  - Config: Fargate, low resources (just HTTP calls) ✓
  - Acceptance: Dispatcher running in ECS ✓
  - ✅ Completed: Task definition JSON, service definition JSON, full Terraform
    module with IAM roles, security groups, auto scaling, CloudWatch logs and
    alarms. CPU: 256, Memory: 512MB, auto-scales 2-10.

- [x] **19.5.1.4** [High/M] Implement async job status API
  - File: `apps/isis/generation-api/src/routes/jobs.ts` ✓
  - Implement: GET /jobs/:id/status endpoint ✓
  - Implement: WebSocket for real-time updates (optional, deferred to future)
  - Acceptance: Clients can poll job status ✓
  - Note: Added JobStatusResponse type, getStatus service method, and route with
    cache control headers. Returns status, progress, queue position, timestamps,
    error (failed), workerId (running).

#### 19.5.2 Result Handling

- [x] **19.5.2.1** [High/M] Implement S3 presigned URL generation
  - File: `libs/shared/storage/src/presigned.ts` ✓
  - Implement: Generate presigned PUT URLs for RunPod uploads ✓
  - Config: 1-hour expiry, correct content-type ✓
  - Acceptance: RunPod can upload directly to S3 ✓
  - ✅ Completed: Constants (DEFAULT_UPLOAD_EXPIRY, MAX_EXPIRY),
    GPU_OUTPUT_CONTENT_TYPES mapping, detectContentType, generateGpuResultKey,
    validateExpiry, generateGpuUploadUrl, generateGpuDownloadUrl,
    generateBatchUploadUrls, createRunPodUploadInstructions. 28 tests.

- [x] **19.5.2.2** [High/M] Implement result callback handler
  - File: `apps/infra/gpu-dispatcher/src/callbacks.ts` ✓
  - Implement: Handle RunPod webhook callbacks ✓
  - Implement: Update job status, trigger downstream events ✓
  - Acceptance: Job completion updates propagate ✓
  - ✅ Completed: CallbackHandler HTTP server with webhook endpoint, HMAC
    signature validation, job mapping system, output extraction, downstream
    callback forwarding with signatures. 32 tests.

- [x] **19.5.2.3** [Medium/M] Implement result validation
  - File: `libs/shared/gpu-dispatcher/src/validation.ts` (new)
  - Implement: Verify uploaded files exist and valid
  - Implement: Check image dimensions, file integrity
  - Acceptance: Invalid results detected and flagged

#### 19.5.3 Error Handling & Resilience

- [x] **19.5.3.1** [High/M] Implement retry logic for RunPod failures
  - File: `libs/shared/gpu-dispatcher/src/retry.ts` (new)
  - Implement: Exponential backoff, max retries
  - Implement: Different retry strategies per error type
  - Acceptance: Transient failures automatically retried
  - ✅ Completed: Multiple strategies (exponential, linear, fibonacci, constant,
    aggressive, conservative), error categorization (transient, rate-limit,
    cold-start, resource, validation, timeout, cancelled), GPU-optimized
    configs, RetryPolicy class, 43 tests

- [x] **19.5.3.2** [High/M] Implement circuit breaker for RunPod
  - File: `libs/shared/gpu-dispatcher/src/circuit-breaker.ts` (new)
  - Implement: Open circuit on repeated failures
  - Implement: Half-open testing, automatic recovery
  - Acceptance: Failing endpoint doesn't cascade

- [x] **19.5.3.3** [High/S] Implement timeout handling
  - File: `libs/shared/gpu-dispatcher/src/timeout.ts` ✓
  - Implement: Job timeout (configurable per type) ✓
  - Implement: Cleanup on timeout ✓
  - Acceptance: Long-running jobs timeout gracefully ✓

- [x] **19.5.3.4** [Medium/M] Implement fallback endpoints
  - File: `libs/shared/gpu-dispatcher/src/fallback.ts` (new)
  - Implement: Try alternative endpoint/GPU if primary fails
  - Acceptance: Increased availability during partial outages

#### 19.5.4 Cost Tracking

- [x] **19.5.4.1** [Medium/M] Implement per-job cost tracking
  - File: `libs/shared/gpu-dispatcher/src/cost-tracker.ts` (new)
  - Implement: Track GPU-seconds per job
  - Implement: Calculate cost based on GPU type and duration
  - Acceptance: Cost per job recorded

- [x] **19.5.4.2** [Medium/M] Create cost dashboard metrics
  - File: `libs/shared/gpu-dispatcher/src/metrics.ts` (new)
  - Implement: Emit metrics for GPU usage, cost, queue depth
  - Acceptance: Metrics visible in CloudWatch
  - ✅ Completed: Prometheus-compatible metrics with job counters, queue gauges,
    duration histograms, cost tracking, endpoint health, circuit breaker state,
    fallback usage. GpuMetricsCollector class for dashboard snapshots. 37 tests
    added, total 205 tests.

---

### 19.6 Model Management

#### 19.6.1 Model Repository

- [x] **19.6.1.1** [High/M] Create S3 bucket for model storage
  - File: `infra/terraform/s3/models.tf` ✓
  - Create: S3 bucket with versioning enabled ✓
  - Config: Lifecycle rules for old versions ✓
  - Acceptance: Bucket created, versioning works ✓
  - Note: Created complete Terraform module with versioning, 5 lifecycle rules
    (version transitions, checkpoints to Glacier, staging expiration, production
    to IA), SSE, public access block, intelligent tiering, optional cross-region
    replication, CloudWatch metrics. Includes dev/prod example tfvars.

- [x] **19.6.1.2** [High/M] Create model registry database schema
  - File: `libs/isis/database/prisma/schema.prisma` ✓ (in domain database)
  - Add: Model table (name, version, s3_path, size, type, metadata) ✓
  - Acceptance: Model metadata stored in database ✓
  - Note: Already implemented as `ModelRegistry` model in isis domain database
    (lines 566-628). Includes: name, version, description, type, format,
    storagePath, sizeBytes, hash, baseModel, trainedOn, triggerWords, tags,
    metadata, compatibleEngines, usage stats, ownership, status, timestamps.

- [x] **19.6.1.3** [High/M] Create model management API
  - File: `apps/isis/generation-api/src/routes/models.ts` ✓
  - Implement: List, upload, download, delete models ✓
  - Implement: Version management ✓
  - Acceptance: Models manageable via API ✓
  - Note: Created full REST API with POST /models (register), GET /models
    (list), GET /models/:id, PUT /models/:id (update), DELETE /models/:id (soft
    delete), GET /models/:id/versions, POST /models/:id/upload-url (presigned
    upload), POST /models/:id/upload-complete (finalize), GET
    /models/:id/download-url, GET /models/by-name/:name/:version. Includes
    model.service.ts (PostgreSQL), model.schema.ts (Zod validation), owner/org
    access control.

#### 19.6.2 Model Caching Strategy

- [x] **19.6.2.1** [High/M] Implement model baking in Docker images
  - Files: `docker/runpod/*/Dockerfile` ✓
  - Config: Common models baked into image layers ✓
  - Config: Use multi-stage builds to manage size ✓
  - Acceptance: Common models available instantly ✓
  - Note: Created multi-stage Docker builds for base and ComfyUI images. Base
    image includes CUDA 12.1.1, Python 3.10, core ML dependencies. ComfyUI image
    has 4 stages: base, models-sd15 (~4GB: SD 1.5, VAE, ControlNets, upscaler),
    models-sdxl (optional ~12GB), runtime. Includes model_utils.py for
    downloading, handler_base.py for RunPod integration.

- [x] **19.6.2.2** [Medium/M] Implement on-demand model download
  - Files: `docker/runpod/*/handler.py` ✓
  - Implement: Download models from S3 on cold start ✓
  - Implement: Cache in /runpod-volume if available ✓
  - Acceptance: Less common models loaded on demand ✓
  - Note: Created on_demand_loader.py with intelligent caching (baked → volume
    cache → S3 → HuggingFace → Registry). Handler extracts model names from job
    input and workflows, loads missing models before execution. Supports
    concurrent downloads, progress callbacks, automatic volume sync.

- **19.6.2.3** Configure RunPod network volume (optional). _Withdrawn
  2026-09-18: network-volume configuration belongs in `desired-state.json`;
  RunPod endpoints are now declared in
  `infra/runpod/endpoints/desired-state.json` and reconciled by
  `.github/workflows/runpod-endpoint-reconcile.yml` (ISIS_CHROMA C.03, checked),
  not clicked in the RunPod console; this item is superseded by that mechanism._

#### 19.6.3 Model Sync Pipeline

- [x] **19.6.3.1** [Medium/M] Create model sync CLI tool
  - File: `tools/model-sync/src/index.ts` ✓
  - Implement: Sync models from S3 to RunPod volume ✓
  - Implement: Diff-based sync, progress reporting ✓
  - Acceptance: Models synced to RunPod storage ✓
  - Note: Created TypeScript CLI with sync, diff, list commands. Features:
    diff-based sync, concurrent downloads, progress bar with ETA, model type
    filtering, include/exclude patterns, checksum verification, dry-run mode,
    orphan detection, JSON output for scripting.

- [x] **19.6.3.2** [Medium/M] Create model update workflow
  - File: `.github/workflows/model-sync.yml` ✓
  - Config: Manual trigger to sync new models ✓
  - Acceptance: New models available on RunPod after sync ✓
  - Note: Created GitHub Actions workflow with analyze, sync, update-registry,
    notify jobs. Supports manual dispatch with model type filtering,
    include/exclude patterns, dry-run, force options. Scheduled weekly sync.
    Creates temporary RunPod pod for sync, auto-cleanup.

---

### 19.7 ComfyUI Migration (RunComfy → RunPod)

#### 19.7.1 Workflow Migration

- **19.7.1.1** Audit existing RunComfy workflows. _Withdrawn 2026-09-18: the
  RunComfy migration was abandoned — RunComfy survives only as a provider enum
  in `libs/yemaya/comfyui-integration`, and ComfyUI-on-RunPod is owned by the
  ISIS_CHROMA track. AWS ECS/EKS were replaced by the single-box Hetzner compose
  plane (`deploy-hetzner.yml`), so this phase's premise is gone._

- **19.7.1.2** Install required ComfyUI custom nodes. _Withdrawn 2026-09-18:
  `docker/runpod/comfyui/` flavor images are slated for retirement (ISIS_CHROMA
  C.12.05); the RunComfy migration is abandoned. AWS ECS/EKS were replaced by
  the single-box Hetzner compose plane (`deploy-hetzner.yml`), so this phase's
  premise is gone._

- **19.7.1.3** Migrate workflow JSON files. _Withdrawn 2026-09-18:
  `assets/comfyui-workflows/` was never created and the RunComfy migration is
  abandoned; ComfyUI-on-RunPod is owned by ISIS_CHROMA._

- **19.7.1.4** Create workflow validation tests. _Withdrawn 2026-09-18: named a
  new `tests/integration/comfyui-workflows.test.ts` (which the conventions
  ratchet would reject as `.test.ts`) and needs a paid GPU endpoint; the
  RunComfy migration is abandoned and ComfyUI validation is owned by
  ISIS_CHROMA._

#### 19.7.2 API Compatibility

- [x] **19.7.2.1** [High/M] Create ComfyUI API adapter ✅
  - File: `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts` (new)
  - Implemented: RunPodAdapter class with same interface as RunComfyProvider
  - Implemented: Job submission, polling, cancellation via RunPod serverless API
  - Implemented: Workflow translation from ComfyUI format to RunPod input
  - Implemented: Status mapping, output mapping, error handling
  - Implemented: Token bucket rate limiter, circuit breaker, retry policy
  - Implemented: Analytics, queue stats, retry stats tracking
  - Implemented: Factory functions (createRunPodAdapter, getRunPodAdapter)
  - Acceptance: Existing code works with minimal changes

- [x] **19.7.2.2** [High/M] Update Isis generation service ✅
  - File: `apps/isis/generation-api/src/services/comfyui.service.ts` (new)
  - Implemented: ComfyUIService with unified interface for ComfyUI execution
  - Implemented: Automatic provider selection based on feature flags
  - Implemented: Failover support between RunComfy and RunPod
  - Implemented: Health monitoring and provider health tracking
  - Implemented: Integration with workflow executor via
    initializeComfyUIService()
  - Implemented: Job statistics, analytics, and queue stats
  - Updated: WorkflowExecutor to use ComfyUIService for real execution
  - Updated: services/index.ts to export ComfyUIService
  - Acceptance: Generation requests route to RunPod

- [x] **19.7.2.3** [Medium/S] Add feature flag for migration ✅
  - File: `libs/shared/config/src/features.ts` (new)
  - Implemented: FeatureFlags registry with all migration flags
  - Implemented: COMFYUI_PROVIDER flag (runcomfy|runpod)
  - Implemented: ENABLE_RUNPOD_MIGRATION with percentage-based routing
  - Implemented: RUNPOD\_\*\_ENDPOINT_ID configuration flags
  - Implemented: shouldUseRunPod(), getComfyUIProvider() helper functions
  - Implemented: validateRunPodConfiguration() for startup validation
  - Config: Image generation, performance, and experimental flags
  - Acceptance: Can switch between providers for testing

---

### 19.8 Observability & Monitoring

#### 19.8.1 CloudWatch Setup

- [x] **19.8.1.1** [High/M] Create CloudWatch log groups ✅
  - File: `infra/terraform/cloudwatch/logs.tf`
  - Implemented: Log groups for all ECS services (flattened from domains)
  - Implemented: Infrastructure log groups (cluster, ALB access, VPC flow)
  - Implemented: RunPod metrics collector log group (conditional)
  - Implemented: Metric filters for errors, warnings, slow requests, HTTP status
    codes
  - Implemented: Business metric filters for Isis generation events
  - Implemented: Log Insights query definitions (error analysis, slow requests,
    etc.)
  - Config: 30-day retention, configurable via variables
  - Acceptance: All ECS logs in CloudWatch

- [x] **19.8.1.2** [High/M] Create CloudWatch dashboards ✅
  - File: `infra/terraform/cloudwatch/dashboards.tf`
  - Implemented: Platform overview dashboard (all services health)
  - Implemented: Per-domain dashboards with CPU, memory, error trends
  - Implemented: RunPod metrics dashboard with GPU utilization, queue depth,
    costs
  - Implemented: Isis generation business metrics dashboard
  - Implemented: Consistent color palette across dashboards
  - Acceptance: Dashboards show key metrics

- [x] **19.8.1.3** [High/M] Create CloudWatch alarms ✅
  - File: `infra/terraform/cloudwatch/alarms.tf`
  - Implemented: Cluster-level alarms (CPU, memory)
  - Implemented: Critical service alarms (CPU, memory, task count)
  - Implemented: Error rate and HTTP 5xx spike alarms per critical domain
  - Implemented: ALB latency (p99) and unhealthy hosts alarms
  - Implemented: RunPod alarms (hourly cost, queue depth, low utilization,
    failure rate)
  - Implemented: Composite alarm for critical services health
  - Config: SNS topic for notifications, configurable thresholds
  - Acceptance: Alarms trigger on threshold breach

#### 19.8.2 RunPod Monitoring

- [x] **19.8.2.1** [High/M] Create RunPod metrics collector ✅
  - File: `apps/infra/runpod-metrics/src/index.ts` (new)
  - Implemented: RunPodMetricsCollector class with health polling
  - Implemented: CloudWatch metric publishing (job counts, worker metrics)
  - Implemented: Calculated metrics (utilization, saturation, failure rate)
  - Implemented: Cost tracking metrics (estimated hourly cost)
  - Implemented: Aggregate metrics across all endpoints
  - Implemented: Configuration loading from environment variables
  - Implemented: Continuous and one-shot (Lambda) run modes
  - Config: tsup build, project.json, package.json
  - Acceptance: RunPod metrics in CloudWatch

- [x] **19.8.2.2** [Medium/M] Add RunPod metrics to dashboard ✅
  - File: `infra/terraform/cloudwatch/dashboards.tf`
  - Implemented: Dedicated RunPod dashboard with full metrics suite
  - Implemented: Workers running, jobs queued, overall utilization gauges
  - Implemented: Estimated hourly cost tracking
  - Implemented: Per-endpoint breakdown (workers, queue, utilization, failure
    rate)
  - Implemented: Jobs completed/failed over time
  - Implemented: Cost by endpoint with alert threshold annotations
  - Acceptance: RunPod metrics visible alongside AWS

- [x] **19.8.2.3** [Medium/M] Create RunPod cost alerts ✅
  - File: `infra/terraform/cloudwatch/alarms.tf`
  - Implemented: Hourly cost threshold alarm
  - Implemented: Queue depth threshold alarm
  - Implemented: Low utilization alarm (cost savings opportunity)
  - Implemented: High failure rate alarm (>10% across endpoints)
  - Config: Configurable thresholds via variables
  - Acceptance: Cost overruns detected quickly

#### 19.8.3 Distributed Tracing

- [x] **19.8.3.1** [Medium/L] Implement X-Ray tracing in ECS services ✅
  - File: `libs/shared/tracing/src/xray.ts` (new)
  - File: `libs/shared/tracing/src/middleware/hono.ts` (new)
  - Implemented: AWS X-Ray trace ID format utilities (parse, format, convert)
  - Implemented: X-Ray tracer configuration for ECS services
  - Implemented: ECS metadata fetching for trace enrichment
  - Implemented: X-Ray annotations and metadata utilities
  - Implemented: Hono tracing middleware (for Isis, Hathor, Yemaya services)
  - Updated: Existing Fastify/Express middleware already available
  - Updated: Package.json with X-Ray propagator dependency
  - Updated: pnpm catalog with @opentelemetry/propagator-aws-xray
  - Config: 5% sampling in production, 100% in development
  - Acceptance: Traces visible in X-Ray console

- [x] **19.8.3.2** [Medium/M] Implement trace propagation to RunPod ✅
  - File: `libs/shared/gpu-dispatcher/src/tracing.ts` (new)
  - Implemented: Trace context injection/extraction for GPU job metadata
  - Implemented: W3C traceparent header propagation to RunPod workers
  - Implemented: GpuJobTracer wrapper for traced dispatch, poll, complete,
    retry, upload
  - Implemented: Span attributes for GPU job lifecycle (type, status, priority,
    endpoint)
  - Implemented: Execution metrics recording in traces
  - Implemented: Job trace link creation for correlation
  - Updated: gpu-dispatcher index.ts with tracing exports
  - Updated: package.json with @oshun/tracing peer dependency
  - Acceptance: End-to-end traces include GPU execution

---

### 19.9 Deployment Pipeline

#### 19.9.1 CI/CD for ECS

- [x] **19.9.1.1** [High/L] Create GitHub Actions workflow for ECS deploy ✅
  - File: `.github/workflows/deploy-ecs.yml` (new)
  - Implemented: Deployment planning with affected service detection
  - Implemented: Build and push to AWS ECR with matrix strategy
  - Implemented: ECS task definition update and service deployment
  - Implemented: Deployment stabilization wait and verification
  - Implemented: Post-deployment health checks
  - Implemented: Deployment summary with Slack notifications
  - Config: Per-environment deployment (staging via main, prod via
    tags/releases)
  - Config: Manual workflow dispatch with domain/service selection
  - Config: Force deploy option for manual runs
  - Acceptance: Commits to main deploy automatically

- [x] **19.9.1.2** [High/M] Implement blue/green deployment ✅
  - File: `infra/terraform/ecs/deployment.tf` (new)
  - Implemented: CodeDeploy application for ECS with blue/green deployment
  - Implemented: CodeDeploy deployment groups per service with traffic routing
  - Implemented: IAM role and policies for CodeDeploy
  - Implemented: Custom deployment configurations (Linear 5%, Canary 5%, Linear
    25%)
  - Implemented: CloudWatch alarms for deployment health (error rate, latency,
    unhealthy hosts)
  - Implemented: Auto-rollback on deployment failure, stop, or alarm
  - Implemented: Optional approval workflow with configurable timeout
  - Implemented: SNS notifications for deployment events
  - Implemented: S3 bucket for AppSpec files (optional)
  - File: `infra/terraform/ecs/variables.tf` - Added blue/green variables
  - File: `infra/terraform/ecs/outputs.tf` - Added blue/green outputs
  - File: `infra/terraform/ecs-services/blue-green.tf` (new)
  - Implemented: Green target groups for blue/green services
  - Implemented: ECS services with CODE_DEPLOY controller
  - Implemented: Auto-scaling for blue/green services
  - File: `infra/terraform/ecs-services/variables.tf` - Added service-level
    blue/green options
  - File: `infra/terraform/ecs-services/main.tf` - Modified for rolling vs
    blue/green separation
  - File: `infra/terraform/ecs-services/outputs.tf` - Added blue/green target
    group outputs
  - Config: Automatic rollback on health check failure
  - Acceptance: Zero-downtime deployments

- [x] **19.9.1.3** [High/M] Create deployment approval workflow ✅
  - File: `.github/workflows/deploy-ecs.yml`
  - Implemented: Separate staging (`deploy-staging`) and production
    (`deploy-production`) jobs
  - Implemented: `production-approval` environment gate requiring manual
    approval
  - Implemented: `approve-production` job blocks until reviewer approval
  - Implemented: Staging deploys automatically (no approval needed)
  - Implemented: Production deploys blocked until approval via GitHub
    Environments
  - Implemented: Approval logging and summary reporting with timestamps
  - Implemented: More conservative deployment settings for production
    (maxPercent=150, failFast=true, maxParallel=2)
  - Implemented: Health check and summary jobs aware of both deployment paths
  - Config: Manual approval required for production via GitHub Environment
    protection rules
  - Note: Requires `production-approval` environment with Required Reviewers
    configured in GitHub repository Settings > Environments
  - Acceptance: Production deploys require human approval

#### 19.9.2 CI/CD for RunPod

- [x] **19.9.2.1** [High/M] Create GitHub Actions workflow for RunPod images ✅
  - File: `.github/workflows/deploy-runpod.yml` (new)
  - Implemented: Build GPU Docker images for RunPod serverless workers
  - Implemented: Automatic change detection via dorny/paths-filter
  - Implemented: Parallel builds for base, comfyui (sd15/sdxl), sd, flux,
    inference
  - Implemented: Base image built first, dependent images follow
  - Implemented: Docker Hub push on main branch merge
  - Implemented: Manual workflow dispatch with image selection
  - Implemented: GitHub Actions cache for faster rebuilds
  - Implemented: Multi-stage build support (ComfyUI sd15/sdxl variants)
  - Implemented: Build summary with status table
  - Implemented: Slack notifications on failure
  - Config: Separate from ECS pipeline
  - Config: Docker Hub namespace `oshunai`
  - Note: Requires DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets
  - Acceptance: RunPod images built and pushed

- [x] **19.9.2.2** [Medium/M] Implement RunPod endpoint update automation ✅
  - File: `.github/workflows/deploy-runpod.yml`
  - Implemented: `update-endpoints` job using RunPod GraphQL API
  - Implemented: Automatic endpoint updates after successful image builds
  - Implemented: Endpoint ID mapping via GitHub repository variables
  - Implemented: Per-endpoint update status tracking
  - Implemented: Error handling with detailed logging
  - Implemented: Endpoint update summary in workflow output
  - Config: Endpoint IDs configured via GitHub vars (RUNPOD*ENDPOINT*\*)
  - Note: Requires RUNPOD_API_KEY secret for authentication
  - Note: RunPod API doesn't support gradual rollout (instant updates)
  - Acceptance: Endpoints updated after image push

#### 19.9.3 Infrastructure as Code

- [x] **19.9.3.1** [High/M] Create Terraform state backend ✅
  - Directory: `infra/terraform/state-backend/` (new module)
  - File: `infra/terraform/state-backend/main.tf` - S3 bucket, DynamoDB, IAM
  - File: `infra/terraform/state-backend/variables.tf` - Configuration options
  - File: `infra/terraform/state-backend/outputs.tf` - Backend config outputs
  - File: `infra/terraform/state-backend/terraform.tfvars.example`
  - Implemented: S3 bucket with versioning and server-side encryption
  - Implemented: DynamoDB table for state locking with PITR
  - Implemented: Optional KMS encryption for enhanced security
  - Implemented: IAM role for CI/CD with GitHub OIDC trust
  - Implemented: Bucket policy enforcing TLS and encryption
  - Implemented: Lifecycle rules for version management
  - Implemented: Auto-generated backend.tf template
  - Implemented: State migration script generator
  - Config: S3 bucket + DynamoDB for state locking
  - Acceptance: Terraform state stored remotely

- [x] **19.9.3.2** [High/M] Create Terraform CI workflow ✅
  - File: `.github/workflows/terraform.yml` (new)
  - Implemented: Change detection for Terraform modules using paths-filter
  - Implemented: Terraform format check (`terraform fmt -check`)
  - Implemented: Terraform validate for configuration syntax
  - Implemented: Plan jobs for staging and production environments
  - Implemented: Apply jobs with staging auto-apply on merge
  - Implemented: Production apply requires approval via GitHub Environment
  - Implemented: Plan output as PR comments for review
  - Implemented: Plan artifacts uploaded for apply job to use
  - Implemented: Workflow summary with module statuses
  - Implemented: Optional Slack notifications for apply results
  - Config: Plan on PR, apply on merge to main
  - Note: Requires `production` environment with Required Reviewers in GitHub
  - Acceptance: Infrastructure changes reviewed and applied

- [x] **19.9.3.3** [Medium/M] Create environment separation ✅
  - Directory: `infra/terraform/environments/` (new)
  - File: `infra/terraform/environments/README.md` - Usage documentation
  - Files: `infra/terraform/environments/staging/` - Staging environment
  - Files: `infra/terraform/environments/production/` - Production environment
  - Implemented: Separate backend.tf with environment-specific state keys
  - Implemented: main.tf referencing all infrastructure modules
  - Implemented: variables.tf with environment-specific defaults
  - Implemented: outputs.tf for environment resources
  - Implemented: terraform.tfvars.example with environment values
  - Staging config: 2 AZs, single NAT, smaller instances, no deletion protection
  - Production config: 3 AZs, HA NAT, larger instances, deletion protection
  - Production config: Autoscaling enabled, KMS encryption, longer retention
  - Config: Separate VPC CIDRs to allow VPC peering if needed
  - Acceptance: Environments isolated

---

### 19.10 Decommissioning Old Infrastructure

#### 19.10.1 Traffic Migration

- **19.10.1.1** Configure DNS for gradual cutover (Route 53 weighted routing to
  ECS). _Withdrawn 2026-09-18: ECS was retired; there is no ECS to cut traffic
  over to. Hetzner DNS is a single A record at the owner's registrar._

- **19.10.1.2** Monitor error rates during cutover (CloudWatch). _Withdrawn
  2026-09-18: no ECS cutover happens; the Hetzner plane's monitoring is
  Prometheus/Grafana in `infra/hetzner/observability`._

- **19.10.1.3** Complete DNS cutover (100% traffic to ECS). _Withdrawn
  2026-09-18: ECS was retired; there is no ECS target._

#### 19.10.2 EKS Decommissioning

- **19.10.2.1** Drain EKS workloads. _Withdrawn 2026-09-18: EKS was never the
  production plane and is retired; there is nothing to drain._

- **19.10.2.2** Delete EKS cluster. _Withdrawn 2026-09-18: no EKS cluster exists
  on the Hetzner plane._

- **19.10.2.3** Clean up EKS IAM roles. _Withdrawn 2026-09-18: no EKS/AWS IAM
  exists on the Hetzner plane._

#### 19.10.3 RunComfy Decommissioning

- **19.10.3.1** Verify all workflows migrated. _Withdrawn 2026-09-18: the
  RunComfy migration was abandoned; endpoint state is reconciled from
  `desired-state.json` (ISIS_CHROMA C.03)._

- [ ] **19.10.3.2** Cancel the RunComfy subscription to stop charges. An account
      action only the owner can take (stated 2026-09-18); RunComfy is no longer
      used (only a provider enum remains). `blocked:external`

- [ ] **19.10.3.3** Export any remaining data stored on RunComfy before
      cancelling. Needs the owner's RunComfy account access to export (stated
      2026-09-18). `blocked:external`

#### 19.10.4 Cost Verification

- **19.10.4.1** Compare before/after costs (AWS and RunPod bills). _Withdrawn
  2026-09-18: there is no AWS/ECS bill — the Hetzner plane is a flat single-box
  cost; a Hetzner-vs-RunPod review is a different, owner-run finance task._

- [ ] **19.10.4.2** Optimize infrastructure based on actual usage (right-size
      the Hetzner box and RunPod scaling). Needs real production traffic to
      right-size against (stated 2026-09-18). `blocked:release`

---

### 19.11 Documentation & Runbooks

#### 19.11.1 Architecture Documentation

- [x] **19.11.1.1** [Medium/M] Document ECS architecture ✅
  - File: `docs/infrastructure/ecs-architecture.md` (new)
  - Content: Cluster setup, services, networking, scaling, monitoring, security
  - Content: Deployment strategies, health checks, troubleshooting
  - Acceptance: New team members can understand setup

- [x] **19.11.1.2** [Medium/M] Document RunPod integration ✅
  - File: `docs/infrastructure/runpod-integration.md` (new)
  - Content: Endpoints configuration, Docker images, handler code
  - Content: API integration, model management, monitoring, cost management
  - Acceptance: GPU workflow documented

- [x] **19.11.1.3** [Medium/M] Document deployment process ✅
  - File: `docs/infrastructure/deployment.md` (new)
  - Content: CI/CD pipelines, deployment stages, strategies
  - Content: Rollback procedures, health checks, best practices
  - Acceptance: Anyone can deploy

#### 19.11.2 Operational Runbooks

- [x] **19.11.2.1** [High/M] Create incident response runbook ✅
  - File: `docs/reference/runbooks/incident-response.md` (new)
  - Content: Severity levels, initial response checklist
  - Content: Common incidents diagnosis and resolution
  - Content: Communication templates, post-incident procedures
  - Acceptance: On-call can handle incidents

- [x] **19.11.2.2** [High/M] Create scaling runbook ✅
  - File: `docs/reference/runbooks/scaling.md` (new)
  - Content: ECS auto scaling, manual scaling, scheduled scaling
  - Content: RunPod workers scaling, database and cache scaling
  - Content: Emergency scaling procedures
  - Acceptance: Can handle traffic spikes

- [x] **19.11.2.3** [Medium/M] Create cost management runbook ✅
  - File: `docs/reference/runbooks/cost-management.md` (new)
  - Content: Cost monitoring with AWS Cost Explorer and budgets
  - Content: Optimization strategies for ECS, RDS, S3, RunPod
  - Content: Cost alerts, monthly reviews, emergency cost reduction
  - Acceptance: Can manage cloud costs

---

## Phase 19 Summary Statistics

| Section                                  | Task Count |
| ---------------------------------------- | ---------- |
| 19.1 ECS Cluster Foundation              | 14         |
| 19.2 API Service Migration               | 24         |
| 19.3 Worker Migration (Non-GPU)          | 9          |
| 19.4 RunPod Serverless Setup             | 16         |
| 19.5 AWS ↔ RunPod Integration Layer      | 14         |
| 19.6 Model Management                    | 8          |
| 19.7 ComfyUI Migration                   | 7          |
| 19.8 Observability & Monitoring          | 9          |
| 19.9 Deployment Pipeline                 | 9          |
| 19.10 Decommissioning Old Infrastructure | 10         |
| 19.11 Documentation & Runbooks           | 6          |
| **TOTAL Phase 19**                       | **126**    |

---

## Updated Grand Total

| Phase                                      | Task Count |
| ------------------------------------------ | ---------- |
| Phase 0-14 (Original)                      | 796        |
| Phase 15: Physical Wellness & Fitness      | 503        |
| Phase 16: Video-to-Animation Pipeline      | 346        |
| Phase 17: State-of-the-Art AI Integration  | 424        |
| Phase 18: Production Readiness Remediation | 847        |
| Phase 19: AWS ECS + RunPod Infrastructure  | 126        |
| **GRAND TOTAL**                            | **3,042**  |

---
