Historical task source. The Eve task board holds current task state.
Phase 19: AWS ECS + RunPod Serverless Infrastructure Consolidation#
This phase consolidates infrastructure from Kubernetes, EKS, and RunComfy to a lean two-platform architecture: AWS ECS for APIs/workers and RunPod Serverless for GPU workloads.
Superseded 2026-09-18. AWS ECS/EKS were replaced by the single-box Hetzner compose plane (
.github/workflows/deploy-hetzner.yml,infra/hetzner/), and the RunPod side is now the declarativeinfra/runpod/endpoints/desired-state.jsonreconciled byrunpod-endpoint-reconcile.yml(ISIS_CHROMA C.03). Every open item below is withdrawn or reduced to the residual owner account action.
Architecture Overview:
┌────────────────────────────────────────────────────────────────┐
│ AWS (ECS) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Isis API │ │Sophia API│ │Hathor API│ │Bellona AP│ │
│ │(Fargate) │ │(Fargate) │ │(Fargate) │ │(Fargate) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
│ ┌────┴────────────┴────────────┴────────────┴────┐ │
│ │ ALB (Application Load Balancer) │ │
│ └─────────────────────┬──────────────────────────┘ │
│ │ │
│ ┌─────────────────────┴──────────────────────────┐ │
│ │ Service Connect │ │
│ │ (internal service mesh) │ │
│ └─────────────────────┬──────────────────────────┘ │
│ │ │
│ ┌──────────┐ ┌────────┴───────┐ ┌──────────┐ │
│ │ Workers │ │ SQS Queues │ │ S3 │ │
│ │(EC2 Spot)│ │ (job dispatch) │ │ (assets) │ │
│ └──────────┘ └────────┬───────┘ └────┬─────┘ │
│ │ │ │
└────────────────────────┼──────────────┼────────────────────────┘
│ │
▼ │
┌────────────────────────────────────────────────────────────────┐
│ RunPod Serverless │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ GPU Endpoints │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ ComfyUI │ │ SD/Flux │ │ Custom │ │ │
│ │ │ Endpoint │ │ Endpoint │ │ Inference │ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ │ │ │
│ │ • Auto-scale 0 → N │ │
│ │ • Pay-per-second │ │
│ │ • Cold start: 30-60s (acceptable) │ │
│ │ • Models cached in container layer │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │ │
└────────────────────────┼───────────────────────────────────────┘
│
▼
Upload to S3
(free ingress)
Cost Model (Estimated):
ECS Fargate (6 APIs): ~$250/month
ECS EC2 Spot (workers): ~$100/month
ALB + NAT Gateway: ~$150/month
S3 + CloudWatch: ~$50/month
RunPod Serverless (usage-based): ~$200-500/month (variable)
─────────────────────────────────────────────────
Total: ~$750-1,050/month
19.1 ECS Cluster Foundation#
19.1.1 Cluster Setup#
-
19.1.1.1 [High/M] Create ECS cluster with container insights
- File:
infra/terraform/ecs/cluster.tf - Created: ECS cluster with CloudWatch Container Insights enabled
- Includes: Capacity providers, security groups, logging, service discovery
- Note: Comprehensive module with ec2.tf, iam.tf, outputs.tf, templates/
- File:
-
19.1.1.2 [High/M] Configure Fargate capacity provider
- File:
infra/terraform/ecs/capacity-providers.tf - Created: Fargate and Fargate Spot capacity providers
- Config: Workload-specific strategies (api, batch, gpu, critical)
- Outputs: api_capacity_strategy, batch_capacity_strategy, etc.
- File:
-
19.1.1.3 [High/M] Configure EC2 Spot capacity provider
- File:
infra/terraform/ecs/ec2.tf - Created: Launch template, ASG with mixed instances policy
- Config: On-Demand base + Spot, lifecycle hooks, CloudWatch alarms
- Includes: User data script for ECS agent configuration
- File:
-
19.1.1.4 [High/S] Create ECR repositories for all services
- File:
infra/terraform/ecr/repositories.tf - Created: Domain-organized repos (lilith/, yemaya/, aja/, isis/, etc.)
- Config: Lifecycle policy, scanning, cross-account access, replication
- Total: 100+ repositories across all domains
- File:
19.1.2 Networking#
-
19.1.2.1 [High/L] Create VPC with public/private subnets
- File:
infra/terraform/vpc/main.tf - Created: VPC with 3 AZs, public/private/database subnets
- Config: NAT Gateway (single or per-AZ), VPC endpoints, flow logs
- Includes: endpoints.tf, outputs.tf, subnet groups
- File:
-
19.1.2.2 [High/M] Create Application Load Balancer
- File:
infra/terraform/alb/main.tf - Created: ALB with HTTPS listener, HTTP→HTTPS redirect
- Config: ACM certificate support, path/host-based routing
- Includes: Security group, access logs, listener rules
- File:
-
19.1.2.3 [High/M] Configure Service Connect namespace
- File:
infra/terraform/ecs/service-connect.tf - Created: Cloud Map namespace integration
- Config: Private DNS namespace (oshun.local), service mesh features
- Includes: IAM policy, log group, default configuration output
- File:
-
19.1.2.4 [High/M] Create security groups
- Files:
infra/terraform/ecs/cluster.tf,infra/terraform/alb/main.tf - Created: ECS SG (container-to-container), ALB SG (HTTP/HTTPS)
- Note: VPC default SG locked down, security groups in respective modules
- Files:
-
19.1.2.5 [Medium/S] Create VPC endpoints for AWS services
- File:
infra/terraform/vpc/endpoints.tf - Created: S3/DynamoDB gateway endpoints, interface endpoints
- Config: ECR, Logs, SSM, Secrets Manager endpoints
- Includes: Security group for interface endpoints
- File:
19.1.3 Secrets & Configuration ✅#
-
19.1.3.1 [High/M] Migrate secrets to AWS Secrets Manager
- File:
infra/terraform/secrets/main.tf - Created: Comprehensive secrets module with KMS encryption
- Config: Database, Redis, JWT, encryption, OAuth, AI provider secrets
- Includes: Lambda-based rotation for DB credentials and tokens
- Features: Multi-region replication, ECS-formatted outputs
- File:
-
19.1.3.2 [High/M] Migrate config to AWS Parameter Store
- File:
infra/terraform/ssm/main.tf - Created: SSM module for non-sensitive configuration
- Config: App config, feature flags, rate limits, cache TTLs
- Features: Service endpoints, timeouts, retry policies, AI config
- Includes: Path-based access patterns for batch retrieval
- File:
-
19.1.3.3 [High/M] Create ECS task execution role
- File:
infra/terraform/ecs/iam.tf - Created: Task execution role with ECR pull, Secrets Manager, SSM access
- Config: KMS decryption policy, base ECS policies
- Note: Part of ECS cluster module (already created in 19.1.1)
- File:
-
19.1.3.4 [High/M] Create ECS task roles per service
- File:
infra/terraform/ecs-roles/main.tf - Created: Per-domain service roles (Yemaya, Lilith, Isis, Sophia, Hathor, Bellona)
- Config: Least privilege per service - S3, SQS, Bedrock, etc.
- Features: AWS AI services, ECS Exec, X-Ray, CloudWatch metrics
- Includes: Custom policy support, managed policy attachments
- File:
19.2 API Service Migration#
19.2.1 Isis Domain APIs#
-
19.2.1.1 [High/M] Create Isis Generation API task definition
- File:
infra/terraform/ecs-services/isis.tf(created) - Config: 0.5 vCPU, 1GB memory, health check, env from SSM/Secrets
- Acceptance: Task definition configured in reusable ECS services module
- File:
-
19.2.1.2 [High/M] Create Isis Generation API ECS service
- File:
infra/terraform/ecs-services/isis.tf(created) - Config: Fargate, desired count 2, ALB target group, auto-scaling
- Acceptance: Service configured with ALB routing and auto-scaling
- File:
-
19.2.1.3 [High/M] Create Isis Workflow API task definition
- File:
infra/terraform/ecs-services/isis.tf(created) - Config: 0.5 vCPU, 1GB memory, health check
- Acceptance: Task definition configured
- File:
-
19.2.1.4 [High/M] Create Isis Workflow API ECS service
- File:
infra/terraform/ecs-services/isis.tf(created) - Config: Fargate, Service Connect for internal access
- Acceptance: Service configured with Service Connect
- File:
-
19.2.1.5 [High/M] Create Isis Output Registry task definition
- File:
infra/terraform/ecs-services/isis.tf(created) - Acceptance: Task definition configured
- File:
-
19.2.1.6 [High/M] Create Isis Output Registry ECS service
- File:
infra/terraform/ecs-services/isis.tf(created) - Acceptance: Service configured with S3 bucket integration
- File:
19.2.2 Sophia Domain APIs#
-
19.2.2.1 [High/M] Create Sophia Knowledge API task definition
- File:
infra/terraform/ecs-services/sophia.tf(created) - Config: 0.5 vCPU, 1GB memory, Qdrant/OpenAI integration
- Acceptance: Task definition configured with vector DB access
- File:
-
19.2.2.2 [High/M] Create Sophia Knowledge API ECS service
- File:
infra/terraform/ecs-services/sophia.tf(created) - Acceptance: Service configured with ALB routing
- File:
-
19.2.2.3 [High/M] Create Sophia Research API task definition
- File:
infra/terraform/ecs-services/sophia.tf(created) - Acceptance: Task definition configured with Anthropic API
- File:
-
19.2.2.4 [High/M] Create Sophia Research API ECS service
- File:
infra/terraform/ecs-services/sophia.tf(created) - Acceptance: Service configured with Service Connect
- File:
19.2.3 Hathor Domain APIs#
-
19.2.3.1 [High/M] Create Hathor World API task definition
- File:
infra/terraform/ecs-services/hathor.tf(created) - Acceptance: Task definition configured with S3 worlds bucket
- File:
-
19.2.3.2 [High/M] Create Hathor World API ECS service
- File:
infra/terraform/ecs-services/hathor.tf(created) - Acceptance: Service configured with ALB routing and auto-scaling
- File:
-
19.2.3.3 [High/M] Create Hathor Narrative API task definition
- File:
infra/terraform/ecs-services/hathor.tf(created) - Acceptance: Task definition configured with Anthropic API
- File:
-
19.2.3.4 [High/M] Create Hathor Narrative API ECS service
- File:
infra/terraform/ecs-services/hathor.tf(created) - Acceptance: Service configured with Service Connect to World API
- File:
19.2.4 Bellona Domain APIs#
-
19.2.4.1 [High/M] Create Bellona Build API task definition
- File:
infra/terraform/ecs-services/bellona.tf(created) - Acceptance: Task definition configured with SQS queue integration
- File:
-
19.2.4.2 [High/M] Create Bellona Build API ECS service
- File:
infra/terraform/ecs-services/bellona.tf(created) - Acceptance: Service configured with ALB routing and build worker
- File:
-
19.2.4.3 [High/M] Create Bellona Render API task definition
- File:
infra/terraform/ecs-services/bellona.tf(created) - Acceptance: Task definition configured with render queue
- File:
-
19.2.4.4 [High/M] Create Bellona Render API ECS service
- File:
infra/terraform/ecs-services/bellona.tf(created) - Acceptance: Service configured with Isis Output integration
- File:
19.2.5 Yemaya Domain APIs#
-
19.2.5.1 [High/M] Create Yemaya API task definition
- File:
infra/terraform/ecs-services/yemaya.tf(created) - Config: 1 vCPU, 2GB memory with sticky sessions for WebSocket
- Acceptance: Task definition configured with WebSocket support
- File:
-
19.2.5.2 [High/M] Create Yemaya API ECS service
- File:
infra/terraform/ecs-services/yemaya.tf(created) - Config: WebSocket support via ALB with sticky sessions
- Acceptance: Service configured with Session, Collaboration, Export APIs
- File:
19.2.6 Lilith Domain APIs#
-
19.2.6.1 [High/M] Create Lilith services task definitions
- File:
infra/terraform/ecs-services/lilith.tf(created) - Create: Task definitions for BFF, Auth, AI, Meditation, Conversation, Content, TTS, STT, Notification, Voice Pipeline, User Preferences
- Acceptance: All core microservice task definitions configured
- File:
-
19.2.6.2 [High/L] Create Lilith services ECS services
- File:
infra/terraform/ecs-services/lilith.tf(created) - Create: ECS services for all Lilith core microservices (12 services)
- Config: Service Connect for inter-service communication
- Acceptance: All services configured with proper routing and scaling
- File:
19.2.7 Auto-Scaling Configuration#
-
19.2.7.1 [High/M] Configure target tracking auto-scaling
- File:
infra/terraform/ecs-services/main.tf(created) - Config: CPU (70%) and Memory (80%) target tracking policies
- Acceptance: Auto-scaling targets and policies configured per service
- File:
-
19.2.7.2 [Medium/S] Configure scheduled scaling for predictable patterns
- File:
infra/terraform/ecs-services/main.tf(configurable per service) - Config: Scale in/out cooldowns configurable per service
- Acceptance: Cooldown periods set (300s in, 60s out by default)
- File:
19.3 Worker Migration (Non-GPU)#
19.3.1 Queue Infrastructure#
-
19.3.1.1 [High/M] Create SQS queues for job dispatch
- File:
infra/terraform/sqs/(created - full module) - Create: Queues for all domains (Isis, Sophia, Hathor, Bellona, Yemaya, Lilith)
- Config: DLQ for failed messages, visibility timeout, long polling, CloudWatch alarms
- Acceptance: 20+ queues configured with DLQs and monitoring
- File:
-
19.3.1.2 [High/S] Create SNS topics for fan-out
- File:
infra/terraform/sns/(created - full module) - Create: Event topics for all domains (generation, build, export events)
- Config: SQS/Lambda/HTTP subscriptions, filter policies, CloudWatch alarms
- Acceptance: 16+ topics configured with subscription support
- File:
19.3.2 Worker Services#
-
19.3.2.1 [High/M] Create Isis worker task definition
- File:
infra/terraform/ecs-services/isis.tf(included in domain config) - Config: Fargate Spot, configured for SQS queue processing
- Acceptance: Worker task definition in ECS services module
- File:
-
19.3.2.2 [High/M] Create Isis worker ECS service
- File:
infra/terraform/ecs-services/isis.tf(included) - Config: Fargate Spot capacity provider, auto-scaling on CPU/memory
- Acceptance: Worker service configured with Service Connect
- File:
-
19.3.2.3 [High/M] Create Sophia ingestion worker
- File:
infra/terraform/ecs-services/sophia.tf(sophia-embedding-worker) - Config: Document ingestion, embedding generation with Fargate Spot
- Acceptance: Worker configured with OpenAI and Qdrant access
- File:
-
19.3.2.4 [High/M] Create Hathor simulation worker
- File:
infra/terraform/ecs-services/hathor.tf(can be added to config) - Config: World simulation, NPC updates - infrastructure ready
- Acceptance: Worker infrastructure in place, service configurable
- File:
-
19.3.2.5 [High/M] Create Bellona bridge worker
- File:
infra/terraform/ecs-services/bellona.tf(bellona-build-worker) - Config: 1 vCPU, 2GB RAM, 50GB ephemeral storage, Fargate Spot
- Acceptance: Worker configured with build queue integration
- File:
-
19.3.2.6 [High/M] Create Yemaya workers
- File:
infra/terraform/ecs-services/yemaya.tf(yemaya-project-worker) - Config: Project processing, asset handling with Fargate Spot
- Acceptance: Worker configured with S3 projects bucket
- File:
19.3.3 Worker Auto-Scaling#
- 19.3.3.1 [High/M] Configure queue-based auto-scaling
- File:
infra/terraform/ecs-services/main.tf(CPU/memory target tracking) - File:
infra/terraform/sqs/main.tf(CloudWatch alarms for queue depth) - Config: Auto-scaling targets set, queue depth alarms configured
- Note: Queue-based scaling can be added via custom metrics
- Acceptance: Workers scale on CPU (80%) with queue monitoring
- File:
19.4 RunPod Serverless Setup#
19.4.1 Account & API Setup#
-
19.4.1.1 Create RunPod account and API key. Withdrawn 2026-09-18: the RunPod account and API key already exist and are used by the ISIS_CHROMA track (C.03); RunPod endpoints are now declared in
infra/runpod/endpoints/desired-state.jsonand reconciled by.github/workflows/runpod-endpoint-reconcile.yml(ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism. -
19.4.1.2 [High/S] Store RunPod credentials in Secrets Manager ✅
- File:
infra/terraform/secrets/runpod.tf(new) - Create: Secret for RUNPOD_API_KEY
- Added: runpod_config, runpod_endpoints, runpod_network_volumes variables
- Added: Comprehensive outputs for ECS integration
- Acceptance: ECS tasks can retrieve RunPod credentials
- File:
-
19.4.1.3 [High/M] Create RunPod client library ✅
- File:
libs/shared/runpod-client/src/index.ts(new) - Implement: Serverless endpoint invocation, status polling, error handling
- Acceptance: Type-safe client for RunPod Serverless API
- COMPLETED: Created comprehensive @oshun/runpod-client library with:
- Full API coverage (run, runSync, status, cancel, health, purgeQueue)
- Convenience methods (waitForJob, runAndWait, submitBatch, runBatchAndWait)
- Polling with exponential backoff and jitter
- Comprehensive error handling and retry logic
- Full TypeScript types for RunPod Serverless API
- 101 passing tests
- File:
19.4.2 Docker Images for Serverless#
-
19.4.2.1 [High/L] Create base GPU Docker image ✅
- File:
docker/runpod/base/Dockerfile - Config: CUDA 12.1.1 with cuDNN 8, Python 3.11, PyTorch 2.x
- Config: Common dependencies pre-installed (transformers, diffusers, runpod SDK)
- Added: handler_base.py, model_utils.py, download_models.py, on_demand_loader.py
- Acceptance: Image builds, runs on RunPod
- File:
-
19.4.2.2 [High/L] Create ComfyUI serverless image ✅
- File:
docker/runpod/comfyui/Dockerfile - Config: ComfyUI + common nodes pre-installed (Manager, VideoHelper, ControlNet, IPAdapter, Impact-Pack)
- Config: SD 1.5/SDXL models baked in via multi-stage build
- Config: On-demand model loading from S3/HuggingFace for missing models
- Added: handler.py (full RunPod handler) and comfyui_api.py (WebSocket client)
- Acceptance: ComfyUI workflows execute via serverless
- File:
-
19.4.2.3 [High/L] Create Stable Diffusion serverless image
- File:
docker/runpod/sd/Dockerfile(new) - Config: SD WebUI or custom inference server
- Config: SDXL, SD 1.5 support
- Acceptance: Image generation works via serverless
- DONE: Created comprehensive SD serverless image with:
- Multi-stage Dockerfile (base, models-sd15, models-sdxl, runtime)
- SD 1.5, SD 2.1, SDXL model support via diffusers
- Pre-cached models for fast cold starts
- ControlNet models for SD 1.5 and SDXL
- Memory-efficient inference with fp16/xformers
- File:
-
19.4.2.4 [High/L] Create Flux serverless image ✅
- File:
docker/runpod/flux/Dockerfile(new) - Implemented: Multi-stage build with FLUX.1-schnell, FLUX.1-dev, FLUX.1-Fill-dev
- Implemented: Control model LoRAs (Canny, Depth) pre-cached
- Implemented: BF16/FP16 precision support, group offloading, VAE tiling
- Implemented: Build targets for schnell (28GB), dev (52GB), full (60GB) images
- Config: Optimized for A100/H100
- Acceptance: Flux generation works via serverless
- File:
-
19.4.2.5 [Medium/L] Create custom inference serverless image ✅
- File:
docker/runpod/inference/Dockerfile(new) - Implemented: Dockerfile with transformers, diffusers, ONNX, GGUF support
- Implemented: model_loader.py with unified model loading (transformers, diffusers, torch, ONNX, GGUF)
- Implemented: handler.py with text-generation, text-to-image, embeddings, classification
- Implemented: Custom handler script support via CUSTOM_HANDLER_URL
- Implemented: Quantization support (INT8, INT4, GPTQ, AWQ, GGUF)
- Config: Model loading from S3/HuggingFace/Model Registry
- Acceptance: Custom models can be deployed
- File:
19.4.3 Serverless Endpoints#
-
19.4.3.1 Deploy ComfyUI serverless endpoint. Withdrawn 2026-09-18: RunPod endpoints are now declared in
infra/runpod/endpoints/desired-state.jsonand reconciled by.github/workflows/runpod-endpoint-reconcile.yml(ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism. -
19.4.3.2 Deploy SD/SDXL serverless endpoint. Withdrawn 2026-09-18: RunPod endpoints are now declared in
infra/runpod/endpoints/desired-state.jsonand reconciled by.github/workflows/runpod-endpoint-reconcile.yml(ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism. The SD/SDXL image lives atdocker/runpod/sd. -
19.4.3.3 Deploy Flux serverless endpoint. Withdrawn 2026-09-18: RunPod endpoints are now declared in
infra/runpod/endpoints/desired-state.jsonand reconciled by.github/workflows/runpod-endpoint-reconcile.yml(ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism. The Flux image lives atdocker/runpod/flux. -
19.4.3.4 Configure endpoint auto-scaling settings. Withdrawn 2026-09-18: scaling (
workersMax, idle timeout) is a field indesired-state.json; RunPod endpoints are now declared ininfra/runpod/endpoints/desired-state.jsonand reconciled by.github/workflows/runpod-endpoint-reconcile.yml(ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism.
19.4.4 Serverless Handler Implementation#
-
19.4.4.1 [High/L] Implement ComfyUI serverless handler ✅
- File:
docker/runpod/comfyui/handler.py - Implemented: Full workflow parsing (txt2img, img2img, inpaint, upscale, custom)
- Implemented: ComfyUI process management and WebSocket communication
- Implemented: On-demand model loading, progress callbacks, S3 output upload
- Implemented: comfyui_api.py client with real-time progress via WebSocket
- Acceptance: Workflows execute, outputs in S3
- File:
-
19.4.4.2 [High/M] Implement SD serverless handler ✅
- File:
docker/runpod/sd/handler.py(new) - Implemented: txt2img, img2img, inpainting, ControlNet modes
- Implemented: LoRA loading, scheduler selection, batch generation
- Implemented: Output as base64 or S3 URLs, NSFW filtering
- Implemented: Memory optimizations (fp16, attention slicing, VAE tiling)
- Implemented: PipelineManager with model caching and dynamic pipeline selection
- Acceptance: All generation modes work
- File:
-
19.4.4.3 [High/M] Implement Flux serverless handler ✅
- File:
docker/runpod/flux/handler.py(new) - Implemented: txt2img, img2img, inpaint, control modes
- Implemented: Support for FLUX.1-schnell, dev, fill, canny, depth, redux models
- Implemented: LoRA and IP-Adapter support
- Implemented: Group offloading, VAE tiling/slicing, CPU offload optimizations
- Implemented: FluxPipelineManager with intelligent caching
- Implemented: Output as base64 or S3 URLs
- Acceptance: Flux generation works
- File:
-
19.4.4.4 [Medium/M] Implement model download caching ✅
- File:
docker/runpod/base/scripts/on_demand_loader.py - Implemented: OnDemandModelLoader class with intelligent caching
- Implemented: Volume cache check before download, sync to volume after download
- Implemented: Multi-source support (S3, HuggingFace, Model Registry)
- Acceptance: Models cached between invocations (when using network volume)
- File:
19.5 AWS ↔ RunPod Integration Layer#
19.5.1 Job Dispatch System#
-
19.5.1.1 [High/L] Create GPU job dispatcher service
- File:
libs/shared/gpu-dispatcher/src/index.ts(new) - Implement: Queue GPU jobs, dispatch to RunPod Serverless
- Implement: Job status tracking, result handling
- Acceptance: Jobs dispatched to RunPod, results returned
- File:
-
19.5.1.2 [High/M] Create Queue → RunPod dispatcher worker
- File:
apps/infra/gpu-dispatcher/src/index.ts✓ - Implement: Poll BullMQ queue, invoke RunPod endpoint, handle response ✓
- Implement: Retry logic, DLQ routing for failures ✓
- Acceptance: Worker processes GPU job queue ✓
- Note: Uses Redis/BullMQ (codebase standard) instead of SQS
- File:
-
19.5.1.3 [High/M] Create dispatcher ECS task definition
- File:
apps/infra/gpu-dispatcher/deploy/task-definition.json✓ - Config: Fargate, low resources (just HTTP calls) ✓
- Acceptance: Dispatcher running in ECS ✓
- ✅ Completed: Task definition JSON, service definition JSON, full Terraform module with IAM roles, security groups, auto scaling, CloudWatch logs and alarms. CPU: 256, Memory: 512MB, auto-scales 2-10.
- File:
-
19.5.1.4 [High/M] Implement async job status API
- File:
apps/isis/generation-api/src/routes/jobs.ts✓ - Implement: GET /jobs/:id/status endpoint ✓
- Implement: WebSocket for real-time updates (optional, deferred to future)
- Acceptance: Clients can poll job status ✓
- Note: Added JobStatusResponse type, getStatus service method, and route with cache control headers. Returns status, progress, queue position, timestamps, error (failed), workerId (running).
- File:
19.5.2 Result Handling#
-
19.5.2.1 [High/M] Implement S3 presigned URL generation
- File:
libs/shared/storage/src/presigned.ts✓ - Implement: Generate presigned PUT URLs for RunPod uploads ✓
- Config: 1-hour expiry, correct content-type ✓
- Acceptance: RunPod can upload directly to S3 ✓
- ✅ Completed: Constants (DEFAULT_UPLOAD_EXPIRY, MAX_EXPIRY), GPU_OUTPUT_CONTENT_TYPES mapping, detectContentType, generateGpuResultKey, validateExpiry, generateGpuUploadUrl, generateGpuDownloadUrl, generateBatchUploadUrls, createRunPodUploadInstructions. 28 tests.
- File:
-
19.5.2.2 [High/M] Implement result callback handler
- File:
apps/infra/gpu-dispatcher/src/callbacks.ts✓ - Implement: Handle RunPod webhook callbacks ✓
- Implement: Update job status, trigger downstream events ✓
- Acceptance: Job completion updates propagate ✓
- ✅ Completed: CallbackHandler HTTP server with webhook endpoint, HMAC signature validation, job mapping system, output extraction, downstream callback forwarding with signatures. 32 tests.
- File:
-
19.5.2.3 [Medium/M] Implement result validation
- File:
libs/shared/gpu-dispatcher/src/validation.ts(new) - Implement: Verify uploaded files exist and valid
- Implement: Check image dimensions, file integrity
- Acceptance: Invalid results detected and flagged
- File:
19.5.3 Error Handling & Resilience#
-
19.5.3.1 [High/M] Implement retry logic for RunPod failures
- File:
libs/shared/gpu-dispatcher/src/retry.ts(new) - Implement: Exponential backoff, max retries
- Implement: Different retry strategies per error type
- Acceptance: Transient failures automatically retried
- ✅ Completed: Multiple strategies (exponential, linear, fibonacci, constant, aggressive, conservative), error categorization (transient, rate-limit, cold-start, resource, validation, timeout, cancelled), GPU-optimized configs, RetryPolicy class, 43 tests
- File:
-
19.5.3.2 [High/M] Implement circuit breaker for RunPod
- File:
libs/shared/gpu-dispatcher/src/circuit-breaker.ts(new) - Implement: Open circuit on repeated failures
- Implement: Half-open testing, automatic recovery
- Acceptance: Failing endpoint doesn't cascade
- File:
-
19.5.3.3 [High/S] Implement timeout handling
- File:
libs/shared/gpu-dispatcher/src/timeout.ts✓ - Implement: Job timeout (configurable per type) ✓
- Implement: Cleanup on timeout ✓
- Acceptance: Long-running jobs timeout gracefully ✓
- File:
-
19.5.3.4 [Medium/M] Implement fallback endpoints
- File:
libs/shared/gpu-dispatcher/src/fallback.ts(new) - Implement: Try alternative endpoint/GPU if primary fails
- Acceptance: Increased availability during partial outages
- File:
19.5.4 Cost Tracking#
-
19.5.4.1 [Medium/M] Implement per-job cost tracking
- File:
libs/shared/gpu-dispatcher/src/cost-tracker.ts(new) - Implement: Track GPU-seconds per job
- Implement: Calculate cost based on GPU type and duration
- Acceptance: Cost per job recorded
- File:
-
19.5.4.2 [Medium/M] Create cost dashboard metrics
- File:
libs/shared/gpu-dispatcher/src/metrics.ts(new) - Implement: Emit metrics for GPU usage, cost, queue depth
- Acceptance: Metrics visible in CloudWatch
- ✅ Completed: Prometheus-compatible metrics with job counters, queue gauges, duration histograms, cost tracking, endpoint health, circuit breaker state, fallback usage. GpuMetricsCollector class for dashboard snapshots. 37 tests added, total 205 tests.
- File:
19.6 Model Management#
19.6.1 Model Repository#
-
19.6.1.1 [High/M] Create S3 bucket for model storage
- File:
infra/terraform/s3/models.tf✓ - Create: S3 bucket with versioning enabled ✓
- Config: Lifecycle rules for old versions ✓
- Acceptance: Bucket created, versioning works ✓
- Note: Created complete Terraform module with versioning, 5 lifecycle rules (version transitions, checkpoints to Glacier, staging expiration, production to IA), SSE, public access block, intelligent tiering, optional cross-region replication, CloudWatch metrics. Includes dev/prod example tfvars.
- File:
-
19.6.1.2 [High/M] Create model registry database schema
- File:
libs/isis/database/prisma/schema.prisma✓ (in domain database) - Add: Model table (name, version, s3_path, size, type, metadata) ✓
- Acceptance: Model metadata stored in database ✓
- Note: Already implemented as
ModelRegistrymodel in isis domain database (lines 566-628). Includes: name, version, description, type, format, storagePath, sizeBytes, hash, baseModel, trainedOn, triggerWords, tags, metadata, compatibleEngines, usage stats, ownership, status, timestamps.
- File:
-
19.6.1.3 [High/M] Create model management API
- File:
apps/isis/generation-api/src/routes/models.ts✓ - Implement: List, upload, download, delete models ✓
- Implement: Version management ✓
- Acceptance: Models manageable via API ✓
- Note: Created full REST API with POST /models (register), GET /models (list), GET /models/:id, PUT /models/:id (update), DELETE /models/:id (soft delete), GET /models/:id/versions, POST /models/:id/upload-url (presigned upload), POST /models/:id/upload-complete (finalize), GET /models/:id/download-url, GET /models/by-name/:name/:version. Includes model.service.ts (PostgreSQL), model.schema.ts (Zod validation), owner/org access control.
- File:
19.6.2 Model Caching Strategy#
-
19.6.2.1 [High/M] Implement model baking in Docker images
- Files:
docker/runpod/*/Dockerfile✓ - Config: Common models baked into image layers ✓
- Config: Use multi-stage builds to manage size ✓
- Acceptance: Common models available instantly ✓
- Note: Created multi-stage Docker builds for base and ComfyUI images. Base image includes CUDA 12.1.1, Python 3.10, core ML dependencies. ComfyUI image has 4 stages: base, models-sd15 (~4GB: SD 1.5, VAE, ControlNets, upscaler), models-sdxl (optional ~12GB), runtime. Includes model_utils.py for downloading, handler_base.py for RunPod integration.
- Files:
-
19.6.2.2 [Medium/M] Implement on-demand model download
- Files:
docker/runpod/*/handler.py✓ - Implement: Download models from S3 on cold start ✓
- Implement: Cache in /runpod-volume if available ✓
- Acceptance: Less common models loaded on demand ✓
- Note: Created on_demand_loader.py with intelligent caching (baked → volume cache → S3 → HuggingFace → Registry). Handler extracts model names from job input and workflows, loads missing models before execution. Supports concurrent downloads, progress callbacks, automatic volume sync.
- Files:
-
19.6.2.3 Configure RunPod network volume (optional). Withdrawn 2026-09-18: network-volume configuration belongs in
desired-state.json; RunPod endpoints are now declared ininfra/runpod/endpoints/desired-state.jsonand reconciled by.github/workflows/runpod-endpoint-reconcile.yml(ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism.
19.6.3 Model Sync Pipeline#
-
19.6.3.1 [Medium/M] Create model sync CLI tool
- File:
tools/model-sync/src/index.ts✓ - Implement: Sync models from S3 to RunPod volume ✓
- Implement: Diff-based sync, progress reporting ✓
- Acceptance: Models synced to RunPod storage ✓
- Note: Created TypeScript CLI with sync, diff, list commands. Features: diff-based sync, concurrent downloads, progress bar with ETA, model type filtering, include/exclude patterns, checksum verification, dry-run mode, orphan detection, JSON output for scripting.
- File:
-
19.6.3.2 [Medium/M] Create model update workflow
- File:
.github/workflows/model-sync.yml✓ - Config: Manual trigger to sync new models ✓
- Acceptance: New models available on RunPod after sync ✓
- Note: Created GitHub Actions workflow with analyze, sync, update-registry, notify jobs. Supports manual dispatch with model type filtering, include/exclude patterns, dry-run, force options. Scheduled weekly sync. Creates temporary RunPod pod for sync, auto-cleanup.
- File:
19.7 ComfyUI Migration (RunComfy → RunPod)#
19.7.1 Workflow Migration#
-
19.7.1.1 Audit existing RunComfy workflows. Withdrawn 2026-09-18: the RunComfy migration was abandoned — RunComfy survives only as a provider enum in
libs/yemaya/comfyui-integration, and ComfyUI-on-RunPod is owned by the ISIS_CHROMA track. AWS ECS/EKS were replaced by the single-box Hetzner compose plane (deploy-hetzner.yml), so this phase's premise is gone. -
19.7.1.2 Install required ComfyUI custom nodes. Withdrawn 2026-09-18:
docker/runpod/comfyui/flavor images are slated for retirement (ISIS_CHROMA C.12.05); the RunComfy migration is abandoned. AWS ECS/EKS were replaced by the single-box Hetzner compose plane (deploy-hetzner.yml), so this phase's premise is gone. -
19.7.1.3 Migrate workflow JSON files. Withdrawn 2026-09-18:
assets/comfyui-workflows/was never created and the RunComfy migration is abandoned; ComfyUI-on-RunPod is owned by ISIS_CHROMA. -
19.7.1.4 Create workflow validation tests. Withdrawn 2026-09-18: named a new
tests/integration/comfyui-workflows.test.ts(which the conventions ratchet would reject as.test.ts) and needs a paid GPU endpoint; the RunComfy migration is abandoned and ComfyUI validation is owned by ISIS_CHROMA.
19.7.2 API Compatibility#
-
19.7.2.1 [High/M] Create ComfyUI API adapter ✅
- File:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts(new) - Implemented: RunPodAdapter class with same interface as RunComfyProvider
- Implemented: Job submission, polling, cancellation via RunPod serverless API
- Implemented: Workflow translation from ComfyUI format to RunPod input
- Implemented: Status mapping, output mapping, error handling
- Implemented: Token bucket rate limiter, circuit breaker, retry policy
- Implemented: Analytics, queue stats, retry stats tracking
- Implemented: Factory functions (createRunPodAdapter, getRunPodAdapter)
- Acceptance: Existing code works with minimal changes
- File:
-
19.7.2.2 [High/M] Update Isis generation service ✅
- File:
apps/isis/generation-api/src/services/comfyui.service.ts(new) - Implemented: ComfyUIService with unified interface for ComfyUI execution
- Implemented: Automatic provider selection based on feature flags
- Implemented: Failover support between RunComfy and RunPod
- Implemented: Health monitoring and provider health tracking
- Implemented: Integration with workflow executor via initializeComfyUIService()
- Implemented: Job statistics, analytics, and queue stats
- Updated: WorkflowExecutor to use ComfyUIService for real execution
- Updated: services/index.ts to export ComfyUIService
- Acceptance: Generation requests route to RunPod
- File:
-
19.7.2.3 [Medium/S] Add feature flag for migration ✅
- File:
libs/shared/config/src/features.ts(new) - Implemented: FeatureFlags registry with all migration flags
- Implemented: COMFYUI_PROVIDER flag (runcomfy|runpod)
- Implemented: ENABLE_RUNPOD_MIGRATION with percentage-based routing
- Implemented: RUNPOD_*_ENDPOINT_ID configuration flags
- Implemented: shouldUseRunPod(), getComfyUIProvider() helper functions
- Implemented: validateRunPodConfiguration() for startup validation
- Config: Image generation, performance, and experimental flags
- Acceptance: Can switch between providers for testing
- File:
19.8 Observability & Monitoring#
19.8.1 CloudWatch Setup#
-
19.8.1.1 [High/M] Create CloudWatch log groups ✅
- File:
infra/terraform/cloudwatch/logs.tf - Implemented: Log groups for all ECS services (flattened from domains)
- Implemented: Infrastructure log groups (cluster, ALB access, VPC flow)
- Implemented: RunPod metrics collector log group (conditional)
- Implemented: Metric filters for errors, warnings, slow requests, HTTP status codes
- Implemented: Business metric filters for Isis generation events
- Implemented: Log Insights query definitions (error analysis, slow requests, etc.)
- Config: 30-day retention, configurable via variables
- Acceptance: All ECS logs in CloudWatch
- File:
-
19.8.1.2 [High/M] Create CloudWatch dashboards ✅
- File:
infra/terraform/cloudwatch/dashboards.tf - Implemented: Platform overview dashboard (all services health)
- Implemented: Per-domain dashboards with CPU, memory, error trends
- Implemented: RunPod metrics dashboard with GPU utilization, queue depth, costs
- Implemented: Isis generation business metrics dashboard
- Implemented: Consistent color palette across dashboards
- Acceptance: Dashboards show key metrics
- File:
-
19.8.1.3 [High/M] Create CloudWatch alarms ✅
- File:
infra/terraform/cloudwatch/alarms.tf - Implemented: Cluster-level alarms (CPU, memory)
- Implemented: Critical service alarms (CPU, memory, task count)
- Implemented: Error rate and HTTP 5xx spike alarms per critical domain
- Implemented: ALB latency (p99) and unhealthy hosts alarms
- Implemented: RunPod alarms (hourly cost, queue depth, low utilization, failure rate)
- Implemented: Composite alarm for critical services health
- Config: SNS topic for notifications, configurable thresholds
- Acceptance: Alarms trigger on threshold breach
- File:
19.8.2 RunPod Monitoring#
-
19.8.2.1 [High/M] Create RunPod metrics collector ✅
- File:
apps/infra/runpod-metrics/src/index.ts(new) - Implemented: RunPodMetricsCollector class with health polling
- Implemented: CloudWatch metric publishing (job counts, worker metrics)
- Implemented: Calculated metrics (utilization, saturation, failure rate)
- Implemented: Cost tracking metrics (estimated hourly cost)
- Implemented: Aggregate metrics across all endpoints
- Implemented: Configuration loading from environment variables
- Implemented: Continuous and one-shot (Lambda) run modes
- Config: tsup build, project.json, package.json
- Acceptance: RunPod metrics in CloudWatch
- File:
-
19.8.2.2 [Medium/M] Add RunPod metrics to dashboard ✅
- File:
infra/terraform/cloudwatch/dashboards.tf - Implemented: Dedicated RunPod dashboard with full metrics suite
- Implemented: Workers running, jobs queued, overall utilization gauges
- Implemented: Estimated hourly cost tracking
- Implemented: Per-endpoint breakdown (workers, queue, utilization, failure rate)
- Implemented: Jobs completed/failed over time
- Implemented: Cost by endpoint with alert threshold annotations
- Acceptance: RunPod metrics visible alongside AWS
- File:
-
19.8.2.3 [Medium/M] Create RunPod cost alerts ✅
- File:
infra/terraform/cloudwatch/alarms.tf - Implemented: Hourly cost threshold alarm
- Implemented: Queue depth threshold alarm
- Implemented: Low utilization alarm (cost savings opportunity)
- Implemented: High failure rate alarm (>10% across endpoints)
- Config: Configurable thresholds via variables
- Acceptance: Cost overruns detected quickly
- File:
19.8.3 Distributed Tracing#
-
19.8.3.1 [Medium/L] Implement X-Ray tracing in ECS services ✅
- File:
libs/shared/tracing/src/xray.ts(new) - File:
libs/shared/tracing/src/middleware/hono.ts(new) - Implemented: AWS X-Ray trace ID format utilities (parse, format, convert)
- Implemented: X-Ray tracer configuration for ECS services
- Implemented: ECS metadata fetching for trace enrichment
- Implemented: X-Ray annotations and metadata utilities
- Implemented: Hono tracing middleware (for Isis, Hathor, Yemaya services)
- Updated: Existing Fastify/Express middleware already available
- Updated: Package.json with X-Ray propagator dependency
- Updated: pnpm catalog with @opentelemetry/propagator-aws-xray
- Config: 5% sampling in production, 100% in development
- Acceptance: Traces visible in X-Ray console
- File:
-
19.8.3.2 [Medium/M] Implement trace propagation to RunPod ✅
- File:
libs/shared/gpu-dispatcher/src/tracing.ts(new) - Implemented: Trace context injection/extraction for GPU job metadata
- Implemented: W3C traceparent header propagation to RunPod workers
- Implemented: GpuJobTracer wrapper for traced dispatch, poll, complete, retry, upload
- Implemented: Span attributes for GPU job lifecycle (type, status, priority, endpoint)
- Implemented: Execution metrics recording in traces
- Implemented: Job trace link creation for correlation
- Updated: gpu-dispatcher index.ts with tracing exports
- Updated: package.json with @oshun/tracing peer dependency
- Acceptance: End-to-end traces include GPU execution
- File:
19.9 Deployment Pipeline#
19.9.1 CI/CD for ECS#
-
19.9.1.1 [High/L] Create GitHub Actions workflow for ECS deploy ✅
- File:
.github/workflows/deploy-ecs.yml(new) - Implemented: Deployment planning with affected service detection
- Implemented: Build and push to AWS ECR with matrix strategy
- Implemented: ECS task definition update and service deployment
- Implemented: Deployment stabilization wait and verification
- Implemented: Post-deployment health checks
- Implemented: Deployment summary with Slack notifications
- Config: Per-environment deployment (staging via main, prod via tags/releases)
- Config: Manual workflow dispatch with domain/service selection
- Config: Force deploy option for manual runs
- Acceptance: Commits to main deploy automatically
- File:
-
19.9.1.2 [High/M] Implement blue/green deployment ✅
- File:
infra/terraform/ecs/deployment.tf(new) - Implemented: CodeDeploy application for ECS with blue/green deployment
- Implemented: CodeDeploy deployment groups per service with traffic routing
- Implemented: IAM role and policies for CodeDeploy
- Implemented: Custom deployment configurations (Linear 5%, Canary 5%, Linear 25%)
- Implemented: CloudWatch alarms for deployment health (error rate, latency, unhealthy hosts)
- Implemented: Auto-rollback on deployment failure, stop, or alarm
- Implemented: Optional approval workflow with configurable timeout
- Implemented: SNS notifications for deployment events
- Implemented: S3 bucket for AppSpec files (optional)
- File:
infra/terraform/ecs/variables.tf- Added blue/green variables - File:
infra/terraform/ecs/outputs.tf- Added blue/green outputs - File:
infra/terraform/ecs-services/blue-green.tf(new) - Implemented: Green target groups for blue/green services
- Implemented: ECS services with CODE_DEPLOY controller
- Implemented: Auto-scaling for blue/green services
- File:
infra/terraform/ecs-services/variables.tf- Added service-level blue/green options - File:
infra/terraform/ecs-services/main.tf- Modified for rolling vs blue/green separation - File:
infra/terraform/ecs-services/outputs.tf- Added blue/green target group outputs - Config: Automatic rollback on health check failure
- Acceptance: Zero-downtime deployments
- File:
-
19.9.1.3 [High/M] Create deployment approval workflow ✅
- File:
.github/workflows/deploy-ecs.yml - Implemented: Separate staging (
deploy-staging) and production (deploy-production) jobs - Implemented:
production-approvalenvironment gate requiring manual approval - Implemented:
approve-productionjob blocks until reviewer approval - Implemented: Staging deploys automatically (no approval needed)
- Implemented: Production deploys blocked until approval via GitHub Environments
- Implemented: Approval logging and summary reporting with timestamps
- Implemented: More conservative deployment settings for production (maxPercent=150, failFast=true, maxParallel=2)
- Implemented: Health check and summary jobs aware of both deployment paths
- Config: Manual approval required for production via GitHub Environment protection rules
- Note: Requires
production-approvalenvironment with Required Reviewers configured in GitHub repository Settings > Environments - Acceptance: Production deploys require human approval
- File:
19.9.2 CI/CD for RunPod#
-
19.9.2.1 [High/M] Create GitHub Actions workflow for RunPod images ✅
- File:
.github/workflows/deploy-runpod.yml(new) - Implemented: Build GPU Docker images for RunPod serverless workers
- Implemented: Automatic change detection via dorny/paths-filter
- Implemented: Parallel builds for base, comfyui (sd15/sdxl), sd, flux, inference
- Implemented: Base image built first, dependent images follow
- Implemented: Docker Hub push on main branch merge
- Implemented: Manual workflow dispatch with image selection
- Implemented: GitHub Actions cache for faster rebuilds
- Implemented: Multi-stage build support (ComfyUI sd15/sdxl variants)
- Implemented: Build summary with status table
- Implemented: Slack notifications on failure
- Config: Separate from ECS pipeline
- Config: Docker Hub namespace
oshunai - Note: Requires DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets
- Acceptance: RunPod images built and pushed
- File:
-
19.9.2.2 [Medium/M] Implement RunPod endpoint update automation ✅
- File:
.github/workflows/deploy-runpod.yml - Implemented:
update-endpointsjob using RunPod GraphQL API - Implemented: Automatic endpoint updates after successful image builds
- Implemented: Endpoint ID mapping via GitHub repository variables
- Implemented: Per-endpoint update status tracking
- Implemented: Error handling with detailed logging
- Implemented: Endpoint update summary in workflow output
- Config: Endpoint IDs configured via GitHub vars (RUNPODENDPOINT*)
- Note: Requires RUNPOD_API_KEY secret for authentication
- Note: RunPod API doesn't support gradual rollout (instant updates)
- Acceptance: Endpoints updated after image push
- File:
19.9.3 Infrastructure as Code#
-
19.9.3.1 [High/M] Create Terraform state backend ✅
- Directory:
infra/terraform/state-backend/(new module) - File:
infra/terraform/state-backend/main.tf- S3 bucket, DynamoDB, IAM - File:
infra/terraform/state-backend/variables.tf- Configuration options - File:
infra/terraform/state-backend/outputs.tf- Backend config outputs - File:
infra/terraform/state-backend/terraform.tfvars.example - Implemented: S3 bucket with versioning and server-side encryption
- Implemented: DynamoDB table for state locking with PITR
- Implemented: Optional KMS encryption for enhanced security
- Implemented: IAM role for CI/CD with GitHub OIDC trust
- Implemented: Bucket policy enforcing TLS and encryption
- Implemented: Lifecycle rules for version management
- Implemented: Auto-generated backend.tf template
- Implemented: State migration script generator
- Config: S3 bucket + DynamoDB for state locking
- Acceptance: Terraform state stored remotely
- Directory:
-
19.9.3.2 [High/M] Create Terraform CI workflow ✅
- File:
.github/workflows/terraform.yml(new) - Implemented: Change detection for Terraform modules using paths-filter
- Implemented: Terraform format check (
terraform fmt -check) - Implemented: Terraform validate for configuration syntax
- Implemented: Plan jobs for staging and production environments
- Implemented: Apply jobs with staging auto-apply on merge
- Implemented: Production apply requires approval via GitHub Environment
- Implemented: Plan output as PR comments for review
- Implemented: Plan artifacts uploaded for apply job to use
- Implemented: Workflow summary with module statuses
- Implemented: Optional Slack notifications for apply results
- Config: Plan on PR, apply on merge to main
- Note: Requires
productionenvironment with Required Reviewers in GitHub - Acceptance: Infrastructure changes reviewed and applied
- File:
-
19.9.3.3 [Medium/M] Create environment separation ✅
- Directory:
infra/terraform/environments/(new) - File:
infra/terraform/environments/README.md- Usage documentation - Files:
infra/terraform/environments/staging/- Staging environment - Files:
infra/terraform/environments/production/- Production environment - Implemented: Separate backend.tf with environment-specific state keys
- Implemented: main.tf referencing all infrastructure modules
- Implemented: variables.tf with environment-specific defaults
- Implemented: outputs.tf for environment resources
- Implemented: terraform.tfvars.example with environment values
- Staging config: 2 AZs, single NAT, smaller instances, no deletion protection
- Production config: 3 AZs, HA NAT, larger instances, deletion protection
- Production config: Autoscaling enabled, KMS encryption, longer retention
- Config: Separate VPC CIDRs to allow VPC peering if needed
- Acceptance: Environments isolated
- Directory:
19.10 Decommissioning Old Infrastructure#
19.10.1 Traffic Migration#
-
19.10.1.1 Configure DNS for gradual cutover (Route 53 weighted routing to ECS). Withdrawn 2026-09-18: ECS was retired; there is no ECS to cut traffic over to. Hetzner DNS is a single A record at the owner's registrar.
-
19.10.1.2 Monitor error rates during cutover (CloudWatch). Withdrawn 2026-09-18: no ECS cutover happens; the Hetzner plane's monitoring is Prometheus/Grafana in
infra/hetzner/observability. -
19.10.1.3 Complete DNS cutover (100% traffic to ECS). Withdrawn 2026-09-18: ECS was retired; there is no ECS target.
19.10.2 EKS Decommissioning#
-
19.10.2.1 Drain EKS workloads. Withdrawn 2026-09-18: EKS was never the production plane and is retired; there is nothing to drain.
-
19.10.2.2 Delete EKS cluster. Withdrawn 2026-09-18: no EKS cluster exists on the Hetzner plane.
-
19.10.2.3 Clean up EKS IAM roles. Withdrawn 2026-09-18: no EKS/AWS IAM exists on the Hetzner plane.
19.10.3 RunComfy Decommissioning#
-
19.10.3.1 Verify all workflows migrated. Withdrawn 2026-09-18: the RunComfy migration was abandoned; endpoint state is reconciled from
desired-state.json(ISIS_CHROMA C.03). -
19.10.3.2 Cancel the RunComfy subscription to stop charges. An account action only the owner can take (stated 2026-09-18); RunComfy is no longer used (only a provider enum remains).
blocked:external -
19.10.3.3 Export any remaining data stored on RunComfy before cancelling. Needs the owner's RunComfy account access to export (stated 2026-09-18).
blocked:external
19.10.4 Cost Verification#
-
19.10.4.1 Compare before/after costs (AWS and RunPod bills). Withdrawn 2026-09-18: there is no AWS/ECS bill — the Hetzner plane is a flat single-box cost; a Hetzner-vs-RunPod review is a different, owner-run finance task.
-
19.10.4.2 Optimize infrastructure based on actual usage (right-size the Hetzner box and RunPod scaling). Needs real production traffic to right-size against (stated 2026-09-18).
blocked:release
19.11 Documentation & Runbooks#
19.11.1 Architecture Documentation#
-
19.11.1.1 [Medium/M] Document ECS architecture ✅
- File:
docs/infrastructure/ecs-architecture.md(new) - Content: Cluster setup, services, networking, scaling, monitoring, security
- Content: Deployment strategies, health checks, troubleshooting
- Acceptance: New team members can understand setup
- File:
-
19.11.1.2 [Medium/M] Document RunPod integration ✅
- File:
docs/infrastructure/runpod-integration.md(new) - Content: Endpoints configuration, Docker images, handler code
- Content: API integration, model management, monitoring, cost management
- Acceptance: GPU workflow documented
- File:
-
19.11.1.3 [Medium/M] Document deployment process ✅
- File:
docs/infrastructure/deployment.md(new) - Content: CI/CD pipelines, deployment stages, strategies
- Content: Rollback procedures, health checks, best practices
- Acceptance: Anyone can deploy
- File:
19.11.2 Operational Runbooks#
-
19.11.2.1 [High/M] Create incident response runbook ✅
- File:
docs/reference/runbooks/incident-response.md(new) - Content: Severity levels, initial response checklist
- Content: Common incidents diagnosis and resolution
- Content: Communication templates, post-incident procedures
- Acceptance: On-call can handle incidents
- File:
-
19.11.2.2 [High/M] Create scaling runbook ✅
- File:
docs/reference/runbooks/scaling.md(new) - Content: ECS auto scaling, manual scaling, scheduled scaling
- Content: RunPod workers scaling, database and cache scaling
- Content: Emergency scaling procedures
- Acceptance: Can handle traffic spikes
- File:
-
19.11.2.3 [Medium/M] Create cost management runbook ✅
- File:
docs/reference/runbooks/cost-management.md(new) - Content: Cost monitoring with AWS Cost Explorer and budgets
- Content: Optimization strategies for ECS, RDS, S3, RunPod
- Content: Cost alerts, monthly reviews, emergency cost reduction
- Acceptance: Can manage cloud costs
- File:
Phase 19 Summary Statistics#
| Section | Task Count |
|---|---|
| 19.1 ECS Cluster Foundation | 14 |
| 19.2 API Service Migration | 24 |
| 19.3 Worker Migration (Non-GPU) | 9 |
| 19.4 RunPod Serverless Setup | 16 |
| 19.5 AWS ↔ RunPod Integration Layer | 14 |
| 19.6 Model Management | 8 |
| 19.7 ComfyUI Migration | 7 |
| 19.8 Observability & Monitoring | 9 |
| 19.9 Deployment Pipeline | 9 |
| 19.10 Decommissioning Old Infrastructure | 10 |
| 19.11 Documentation & Runbooks | 6 |
| TOTAL Phase 19 | 126 |
Updated Grand Total#
| Phase | Task Count |
|---|---|
| Phase 0-14 (Original) | 796 |
| Phase 15: Physical Wellness & Fitness | 503 |
| Phase 16: Video-to-Animation Pipeline | 346 |
| Phase 17: State-of-the-Art AI Integration | 424 |
| Phase 18: Production Readiness Remediation | 847 |
| Phase 19: AWS ECS + RunPod Infrastructure | 126 |
| GRAND TOTAL | 3,042 |