Eve library

phase-19.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 19: AWS ECS + RunPod Serverless Infrastructure Consolidation#

This phase consolidates infrastructure from Kubernetes, EKS, and RunComfy to a lean two-platform architecture: AWS ECS for APIs/workers and RunPod Serverless for GPU workloads.

Superseded 2026-09-18. AWS ECS/EKS were replaced by the single-box Hetzner compose plane (.github/workflows/deploy-hetzner.yml, infra/hetzner/), and the RunPod side is now the declarative infra/runpod/endpoints/desired-state.json reconciled by runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03). Every open item below is withdrawn or reduced to the residual owner account action.

Architecture Overview:

text
┌────────────────────────────────────────────────────────────────┐
│                        AWS (ECS)                               │
│                                                                │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐          │
│  │Isis API  │ │Sophia API│ │Hathor API│ │Bellona AP│          │
│  │(Fargate) │ │(Fargate) │ │(Fargate) │ │(Fargate) │          │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘          │
│       │            │            │            │                 │
│  ┌────┴────────────┴────────────┴────────────┴────┐           │
│  │              ALB (Application Load Balancer)    │           │
│  └─────────────────────┬──────────────────────────┘           │
│                        │                                       │
│  ┌─────────────────────┴──────────────────────────┐           │
│  │                Service Connect                  │           │
│  │            (internal service mesh)              │           │
│  └─────────────────────┬──────────────────────────┘           │
│                        │                                       │
│  ┌──────────┐ ┌────────┴───────┐ ┌──────────┐                 │
│  │ Workers  │ │   SQS Queues   │ │   S3     │                 │
│  │(EC2 Spot)│ │ (job dispatch) │ │ (assets) │                 │
│  └──────────┘ └────────┬───────┘ └────┬─────┘                 │
│                        │              │                        │
└────────────────────────┼──────────────┼────────────────────────┘
                         │              │
                         ▼              │
┌────────────────────────────────────────────────────────────────┐
│                   RunPod Serverless                            │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │                  GPU Endpoints                            │ │
│  │  ┌────────────┐ ┌────────────┐ ┌────────────┐            │ │
│  │  │  ComfyUI   │ │  SD/Flux   │ │  Custom    │            │ │
│  │  │  Endpoint  │ │  Endpoint  │ │  Inference │            │ │
│  │  └────────────┘ └────────────┘ └────────────┘            │ │
│  │                                                           │ │
│  │  • Auto-scale 0 → N                                       │ │
│  │  • Pay-per-second                                         │ │
│  │  • Cold start: 30-60s (acceptable)                        │ │
│  │  • Models cached in container layer                       │ │
│  └──────────────────────────────────────────────────────────┘ │
│                        │                                       │
└────────────────────────┼───────────────────────────────────────┘
                         │
                         ▼
                    Upload to S3
                   (free ingress)

Cost Model (Estimated):

text
ECS Fargate (6 APIs):              ~$250/month
ECS EC2 Spot (workers):            ~$100/month
ALB + NAT Gateway:                 ~$150/month
S3 + CloudWatch:                   ~$50/month
RunPod Serverless (usage-based):   ~$200-500/month (variable)
─────────────────────────────────────────────────
Total:                             ~$750-1,050/month

19.1 ECS Cluster Foundation#

19.1.1 Cluster Setup#

  • 19.1.1.1 [High/M] Create ECS cluster with container insights

    • File: infra/terraform/ecs/cluster.tf
    • Created: ECS cluster with CloudWatch Container Insights enabled
    • Includes: Capacity providers, security groups, logging, service discovery
    • Note: Comprehensive module with ec2.tf, iam.tf, outputs.tf, templates/
  • 19.1.1.2 [High/M] Configure Fargate capacity provider

    • File: infra/terraform/ecs/capacity-providers.tf
    • Created: Fargate and Fargate Spot capacity providers
    • Config: Workload-specific strategies (api, batch, gpu, critical)
    • Outputs: api_capacity_strategy, batch_capacity_strategy, etc.
  • 19.1.1.3 [High/M] Configure EC2 Spot capacity provider

    • File: infra/terraform/ecs/ec2.tf
    • Created: Launch template, ASG with mixed instances policy
    • Config: On-Demand base + Spot, lifecycle hooks, CloudWatch alarms
    • Includes: User data script for ECS agent configuration
  • 19.1.1.4 [High/S] Create ECR repositories for all services

    • File: infra/terraform/ecr/repositories.tf
    • Created: Domain-organized repos (lilith/, yemaya/, aja/, isis/, etc.)
    • Config: Lifecycle policy, scanning, cross-account access, replication
    • Total: 100+ repositories across all domains

19.1.2 Networking#

  • 19.1.2.1 [High/L] Create VPC with public/private subnets

    • File: infra/terraform/vpc/main.tf
    • Created: VPC with 3 AZs, public/private/database subnets
    • Config: NAT Gateway (single or per-AZ), VPC endpoints, flow logs
    • Includes: endpoints.tf, outputs.tf, subnet groups
  • 19.1.2.2 [High/M] Create Application Load Balancer

    • File: infra/terraform/alb/main.tf
    • Created: ALB with HTTPS listener, HTTP→HTTPS redirect
    • Config: ACM certificate support, path/host-based routing
    • Includes: Security group, access logs, listener rules
  • 19.1.2.3 [High/M] Configure Service Connect namespace

    • File: infra/terraform/ecs/service-connect.tf
    • Created: Cloud Map namespace integration
    • Config: Private DNS namespace (oshun.local), service mesh features
    • Includes: IAM policy, log group, default configuration output
  • 19.1.2.4 [High/M] Create security groups

    • Files: infra/terraform/ecs/cluster.tf, infra/terraform/alb/main.tf
    • Created: ECS SG (container-to-container), ALB SG (HTTP/HTTPS)
    • Note: VPC default SG locked down, security groups in respective modules
  • 19.1.2.5 [Medium/S] Create VPC endpoints for AWS services

    • File: infra/terraform/vpc/endpoints.tf
    • Created: S3/DynamoDB gateway endpoints, interface endpoints
    • Config: ECR, Logs, SSM, Secrets Manager endpoints
    • Includes: Security group for interface endpoints

19.1.3 Secrets & Configuration ✅#

  • 19.1.3.1 [High/M] Migrate secrets to AWS Secrets Manager

    • File: infra/terraform/secrets/main.tf
    • Created: Comprehensive secrets module with KMS encryption
    • Config: Database, Redis, JWT, encryption, OAuth, AI provider secrets
    • Includes: Lambda-based rotation for DB credentials and tokens
    • Features: Multi-region replication, ECS-formatted outputs
  • 19.1.3.2 [High/M] Migrate config to AWS Parameter Store

    • File: infra/terraform/ssm/main.tf
    • Created: SSM module for non-sensitive configuration
    • Config: App config, feature flags, rate limits, cache TTLs
    • Features: Service endpoints, timeouts, retry policies, AI config
    • Includes: Path-based access patterns for batch retrieval
  • 19.1.3.3 [High/M] Create ECS task execution role

    • File: infra/terraform/ecs/iam.tf
    • Created: Task execution role with ECR pull, Secrets Manager, SSM access
    • Config: KMS decryption policy, base ECS policies
    • Note: Part of ECS cluster module (already created in 19.1.1)
  • 19.1.3.4 [High/M] Create ECS task roles per service

    • File: infra/terraform/ecs-roles/main.tf
    • Created: Per-domain service roles (Yemaya, Lilith, Isis, Sophia, Hathor, Bellona)
    • Config: Least privilege per service - S3, SQS, Bedrock, etc.
    • Features: AWS AI services, ECS Exec, X-Ray, CloudWatch metrics
    • Includes: Custom policy support, managed policy attachments

19.2 API Service Migration#

19.2.1 Isis Domain APIs#

  • 19.2.1.1 [High/M] Create Isis Generation API task definition

    • File: infra/terraform/ecs-services/isis.tf (created)
    • Config: 0.5 vCPU, 1GB memory, health check, env from SSM/Secrets
    • Acceptance: Task definition configured in reusable ECS services module
  • 19.2.1.2 [High/M] Create Isis Generation API ECS service

    • File: infra/terraform/ecs-services/isis.tf (created)
    • Config: Fargate, desired count 2, ALB target group, auto-scaling
    • Acceptance: Service configured with ALB routing and auto-scaling
  • 19.2.1.3 [High/M] Create Isis Workflow API task definition

    • File: infra/terraform/ecs-services/isis.tf (created)
    • Config: 0.5 vCPU, 1GB memory, health check
    • Acceptance: Task definition configured
  • 19.2.1.4 [High/M] Create Isis Workflow API ECS service

    • File: infra/terraform/ecs-services/isis.tf (created)
    • Config: Fargate, Service Connect for internal access
    • Acceptance: Service configured with Service Connect
  • 19.2.1.5 [High/M] Create Isis Output Registry task definition

    • File: infra/terraform/ecs-services/isis.tf (created)
    • Acceptance: Task definition configured
  • 19.2.1.6 [High/M] Create Isis Output Registry ECS service

    • File: infra/terraform/ecs-services/isis.tf (created)
    • Acceptance: Service configured with S3 bucket integration

19.2.2 Sophia Domain APIs#

  • 19.2.2.1 [High/M] Create Sophia Knowledge API task definition

    • File: infra/terraform/ecs-services/sophia.tf (created)
    • Config: 0.5 vCPU, 1GB memory, Qdrant/OpenAI integration
    • Acceptance: Task definition configured with vector DB access
  • 19.2.2.2 [High/M] Create Sophia Knowledge API ECS service

    • File: infra/terraform/ecs-services/sophia.tf (created)
    • Acceptance: Service configured with ALB routing
  • 19.2.2.3 [High/M] Create Sophia Research API task definition

    • File: infra/terraform/ecs-services/sophia.tf (created)
    • Acceptance: Task definition configured with Anthropic API
  • 19.2.2.4 [High/M] Create Sophia Research API ECS service

    • File: infra/terraform/ecs-services/sophia.tf (created)
    • Acceptance: Service configured with Service Connect

19.2.3 Hathor Domain APIs#

  • 19.2.3.1 [High/M] Create Hathor World API task definition

    • File: infra/terraform/ecs-services/hathor.tf (created)
    • Acceptance: Task definition configured with S3 worlds bucket
  • 19.2.3.2 [High/M] Create Hathor World API ECS service

    • File: infra/terraform/ecs-services/hathor.tf (created)
    • Acceptance: Service configured with ALB routing and auto-scaling
  • 19.2.3.3 [High/M] Create Hathor Narrative API task definition

    • File: infra/terraform/ecs-services/hathor.tf (created)
    • Acceptance: Task definition configured with Anthropic API
  • 19.2.3.4 [High/M] Create Hathor Narrative API ECS service

    • File: infra/terraform/ecs-services/hathor.tf (created)
    • Acceptance: Service configured with Service Connect to World API

19.2.4 Bellona Domain APIs#

  • 19.2.4.1 [High/M] Create Bellona Build API task definition

    • File: infra/terraform/ecs-services/bellona.tf (created)
    • Acceptance: Task definition configured with SQS queue integration
  • 19.2.4.2 [High/M] Create Bellona Build API ECS service

    • File: infra/terraform/ecs-services/bellona.tf (created)
    • Acceptance: Service configured with ALB routing and build worker
  • 19.2.4.3 [High/M] Create Bellona Render API task definition

    • File: infra/terraform/ecs-services/bellona.tf (created)
    • Acceptance: Task definition configured with render queue
  • 19.2.4.4 [High/M] Create Bellona Render API ECS service

    • File: infra/terraform/ecs-services/bellona.tf (created)
    • Acceptance: Service configured with Isis Output integration

19.2.5 Yemaya Domain APIs#

  • 19.2.5.1 [High/M] Create Yemaya API task definition

    • File: infra/terraform/ecs-services/yemaya.tf (created)
    • Config: 1 vCPU, 2GB memory with sticky sessions for WebSocket
    • Acceptance: Task definition configured with WebSocket support
  • 19.2.5.2 [High/M] Create Yemaya API ECS service

    • File: infra/terraform/ecs-services/yemaya.tf (created)
    • Config: WebSocket support via ALB with sticky sessions
    • Acceptance: Service configured with Session, Collaboration, Export APIs

19.2.6 Lilith Domain APIs#

  • 19.2.6.1 [High/M] Create Lilith services task definitions

    • File: infra/terraform/ecs-services/lilith.tf (created)
    • Create: Task definitions for BFF, Auth, AI, Meditation, Conversation, Content, TTS, STT, Notification, Voice Pipeline, User Preferences
    • Acceptance: All core microservice task definitions configured
  • 19.2.6.2 [High/L] Create Lilith services ECS services

    • File: infra/terraform/ecs-services/lilith.tf (created)
    • Create: ECS services for all Lilith core microservices (12 services)
    • Config: Service Connect for inter-service communication
    • Acceptance: All services configured with proper routing and scaling

19.2.7 Auto-Scaling Configuration#

  • 19.2.7.1 [High/M] Configure target tracking auto-scaling

    • File: infra/terraform/ecs-services/main.tf (created)
    • Config: CPU (70%) and Memory (80%) target tracking policies
    • Acceptance: Auto-scaling targets and policies configured per service
  • 19.2.7.2 [Medium/S] Configure scheduled scaling for predictable patterns

    • File: infra/terraform/ecs-services/main.tf (configurable per service)
    • Config: Scale in/out cooldowns configurable per service
    • Acceptance: Cooldown periods set (300s in, 60s out by default)

19.3 Worker Migration (Non-GPU)#

19.3.1 Queue Infrastructure#

  • 19.3.1.1 [High/M] Create SQS queues for job dispatch

    • File: infra/terraform/sqs/ (created - full module)
    • Create: Queues for all domains (Isis, Sophia, Hathor, Bellona, Yemaya, Lilith)
    • Config: DLQ for failed messages, visibility timeout, long polling, CloudWatch alarms
    • Acceptance: 20+ queues configured with DLQs and monitoring
  • 19.3.1.2 [High/S] Create SNS topics for fan-out

    • File: infra/terraform/sns/ (created - full module)
    • Create: Event topics for all domains (generation, build, export events)
    • Config: SQS/Lambda/HTTP subscriptions, filter policies, CloudWatch alarms
    • Acceptance: 16+ topics configured with subscription support

19.3.2 Worker Services#

  • 19.3.2.1 [High/M] Create Isis worker task definition

    • File: infra/terraform/ecs-services/isis.tf (included in domain config)
    • Config: Fargate Spot, configured for SQS queue processing
    • Acceptance: Worker task definition in ECS services module
  • 19.3.2.2 [High/M] Create Isis worker ECS service

    • File: infra/terraform/ecs-services/isis.tf (included)
    • Config: Fargate Spot capacity provider, auto-scaling on CPU/memory
    • Acceptance: Worker service configured with Service Connect
  • 19.3.2.3 [High/M] Create Sophia ingestion worker

    • File: infra/terraform/ecs-services/sophia.tf (sophia-embedding-worker)
    • Config: Document ingestion, embedding generation with Fargate Spot
    • Acceptance: Worker configured with OpenAI and Qdrant access
  • 19.3.2.4 [High/M] Create Hathor simulation worker

    • File: infra/terraform/ecs-services/hathor.tf (can be added to config)
    • Config: World simulation, NPC updates - infrastructure ready
    • Acceptance: Worker infrastructure in place, service configurable
  • 19.3.2.5 [High/M] Create Bellona bridge worker

    • File: infra/terraform/ecs-services/bellona.tf (bellona-build-worker)
    • Config: 1 vCPU, 2GB RAM, 50GB ephemeral storage, Fargate Spot
    • Acceptance: Worker configured with build queue integration
  • 19.3.2.6 [High/M] Create Yemaya workers

    • File: infra/terraform/ecs-services/yemaya.tf (yemaya-project-worker)
    • Config: Project processing, asset handling with Fargate Spot
    • Acceptance: Worker configured with S3 projects bucket

19.3.3 Worker Auto-Scaling#

  • 19.3.3.1 [High/M] Configure queue-based auto-scaling
    • File: infra/terraform/ecs-services/main.tf (CPU/memory target tracking)
    • File: infra/terraform/sqs/main.tf (CloudWatch alarms for queue depth)
    • Config: Auto-scaling targets set, queue depth alarms configured
    • Note: Queue-based scaling can be added via custom metrics
    • Acceptance: Workers scale on CPU (80%) with queue monitoring

19.4 RunPod Serverless Setup#

19.4.1 Account & API Setup#

  • 19.4.1.1 Create RunPod account and API key. Withdrawn 2026-09-18: the RunPod account and API key already exist and are used by the ISIS_CHROMA track (C.03); RunPod endpoints are now declared in infra/runpod/endpoints/desired-state.json and reconciled by .github/workflows/runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism.

  • 19.4.1.2 [High/S] Store RunPod credentials in Secrets Manager ✅

    • File: infra/terraform/secrets/runpod.tf (new)
    • Create: Secret for RUNPOD_API_KEY
    • Added: runpod_config, runpod_endpoints, runpod_network_volumes variables
    • Added: Comprehensive outputs for ECS integration
    • Acceptance: ECS tasks can retrieve RunPod credentials
  • 19.4.1.3 [High/M] Create RunPod client library ✅

    • File: libs/shared/runpod-client/src/index.ts (new)
    • Implement: Serverless endpoint invocation, status polling, error handling
    • Acceptance: Type-safe client for RunPod Serverless API
    • COMPLETED: Created comprehensive @oshun/runpod-client library with:
      • Full API coverage (run, runSync, status, cancel, health, purgeQueue)
      • Convenience methods (waitForJob, runAndWait, submitBatch, runBatchAndWait)
      • Polling with exponential backoff and jitter
      • Comprehensive error handling and retry logic
      • Full TypeScript types for RunPod Serverless API
      • 101 passing tests

19.4.2 Docker Images for Serverless#

  • 19.4.2.1 [High/L] Create base GPU Docker image ✅

    • File: docker/runpod/base/Dockerfile
    • Config: CUDA 12.1.1 with cuDNN 8, Python 3.11, PyTorch 2.x
    • Config: Common dependencies pre-installed (transformers, diffusers, runpod SDK)
    • Added: handler_base.py, model_utils.py, download_models.py, on_demand_loader.py
    • Acceptance: Image builds, runs on RunPod
  • 19.4.2.2 [High/L] Create ComfyUI serverless image ✅

    • File: docker/runpod/comfyui/Dockerfile
    • Config: ComfyUI + common nodes pre-installed (Manager, VideoHelper, ControlNet, IPAdapter, Impact-Pack)
    • Config: SD 1.5/SDXL models baked in via multi-stage build
    • Config: On-demand model loading from S3/HuggingFace for missing models
    • Added: handler.py (full RunPod handler) and comfyui_api.py (WebSocket client)
    • Acceptance: ComfyUI workflows execute via serverless
  • 19.4.2.3 [High/L] Create Stable Diffusion serverless image

    • File: docker/runpod/sd/Dockerfile (new)
    • Config: SD WebUI or custom inference server
    • Config: SDXL, SD 1.5 support
    • Acceptance: Image generation works via serverless
    • DONE: Created comprehensive SD serverless image with:
      • Multi-stage Dockerfile (base, models-sd15, models-sdxl, runtime)
      • SD 1.5, SD 2.1, SDXL model support via diffusers
      • Pre-cached models for fast cold starts
      • ControlNet models for SD 1.5 and SDXL
      • Memory-efficient inference with fp16/xformers
  • 19.4.2.4 [High/L] Create Flux serverless image ✅

    • File: docker/runpod/flux/Dockerfile (new)
    • Implemented: Multi-stage build with FLUX.1-schnell, FLUX.1-dev, FLUX.1-Fill-dev
    • Implemented: Control model LoRAs (Canny, Depth) pre-cached
    • Implemented: BF16/FP16 precision support, group offloading, VAE tiling
    • Implemented: Build targets for schnell (28GB), dev (52GB), full (60GB) images
    • Config: Optimized for A100/H100
    • Acceptance: Flux generation works via serverless
  • 19.4.2.5 [Medium/L] Create custom inference serverless image ✅

    • File: docker/runpod/inference/Dockerfile (new)
    • Implemented: Dockerfile with transformers, diffusers, ONNX, GGUF support
    • Implemented: model_loader.py with unified model loading (transformers, diffusers, torch, ONNX, GGUF)
    • Implemented: handler.py with text-generation, text-to-image, embeddings, classification
    • Implemented: Custom handler script support via CUSTOM_HANDLER_URL
    • Implemented: Quantization support (INT8, INT4, GPTQ, AWQ, GGUF)
    • Config: Model loading from S3/HuggingFace/Model Registry
    • Acceptance: Custom models can be deployed

19.4.3 Serverless Endpoints#

  • 19.4.3.1 Deploy ComfyUI serverless endpoint. Withdrawn 2026-09-18: RunPod endpoints are now declared in infra/runpod/endpoints/desired-state.json and reconciled by .github/workflows/runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism.

  • 19.4.3.2 Deploy SD/SDXL serverless endpoint. Withdrawn 2026-09-18: RunPod endpoints are now declared in infra/runpod/endpoints/desired-state.json and reconciled by .github/workflows/runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism. The SD/SDXL image lives at docker/runpod/sd.

  • 19.4.3.3 Deploy Flux serverless endpoint. Withdrawn 2026-09-18: RunPod endpoints are now declared in infra/runpod/endpoints/desired-state.json and reconciled by .github/workflows/runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism. The Flux image lives at docker/runpod/flux.

  • 19.4.3.4 Configure endpoint auto-scaling settings. Withdrawn 2026-09-18: scaling (workersMax, idle timeout) is a field in desired-state.json; RunPod endpoints are now declared in infra/runpod/endpoints/desired-state.json and reconciled by .github/workflows/runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism.

19.4.4 Serverless Handler Implementation#

  • 19.4.4.1 [High/L] Implement ComfyUI serverless handler ✅

    • File: docker/runpod/comfyui/handler.py
    • Implemented: Full workflow parsing (txt2img, img2img, inpaint, upscale, custom)
    • Implemented: ComfyUI process management and WebSocket communication
    • Implemented: On-demand model loading, progress callbacks, S3 output upload
    • Implemented: comfyui_api.py client with real-time progress via WebSocket
    • Acceptance: Workflows execute, outputs in S3
  • 19.4.4.2 [High/M] Implement SD serverless handler ✅

    • File: docker/runpod/sd/handler.py (new)
    • Implemented: txt2img, img2img, inpainting, ControlNet modes
    • Implemented: LoRA loading, scheduler selection, batch generation
    • Implemented: Output as base64 or S3 URLs, NSFW filtering
    • Implemented: Memory optimizations (fp16, attention slicing, VAE tiling)
    • Implemented: PipelineManager with model caching and dynamic pipeline selection
    • Acceptance: All generation modes work
  • 19.4.4.3 [High/M] Implement Flux serverless handler ✅

    • File: docker/runpod/flux/handler.py (new)
    • Implemented: txt2img, img2img, inpaint, control modes
    • Implemented: Support for FLUX.1-schnell, dev, fill, canny, depth, redux models
    • Implemented: LoRA and IP-Adapter support
    • Implemented: Group offloading, VAE tiling/slicing, CPU offload optimizations
    • Implemented: FluxPipelineManager with intelligent caching
    • Implemented: Output as base64 or S3 URLs
    • Acceptance: Flux generation works
  • 19.4.4.4 [Medium/M] Implement model download caching ✅

    • File: docker/runpod/base/scripts/on_demand_loader.py
    • Implemented: OnDemandModelLoader class with intelligent caching
    • Implemented: Volume cache check before download, sync to volume after download
    • Implemented: Multi-source support (S3, HuggingFace, Model Registry)
    • Acceptance: Models cached between invocations (when using network volume)

19.5 AWS ↔ RunPod Integration Layer#

19.5.1 Job Dispatch System#

  • 19.5.1.1 [High/L] Create GPU job dispatcher service

    • File: libs/shared/gpu-dispatcher/src/index.ts (new)
    • Implement: Queue GPU jobs, dispatch to RunPod Serverless
    • Implement: Job status tracking, result handling
    • Acceptance: Jobs dispatched to RunPod, results returned
  • 19.5.1.2 [High/M] Create Queue → RunPod dispatcher worker

    • File: apps/infra/gpu-dispatcher/src/index.ts ✓
    • Implement: Poll BullMQ queue, invoke RunPod endpoint, handle response ✓
    • Implement: Retry logic, DLQ routing for failures ✓
    • Acceptance: Worker processes GPU job queue ✓
    • Note: Uses Redis/BullMQ (codebase standard) instead of SQS
  • 19.5.1.3 [High/M] Create dispatcher ECS task definition

    • File: apps/infra/gpu-dispatcher/deploy/task-definition.json ✓
    • Config: Fargate, low resources (just HTTP calls) ✓
    • Acceptance: Dispatcher running in ECS ✓
    • ✅ Completed: Task definition JSON, service definition JSON, full Terraform module with IAM roles, security groups, auto scaling, CloudWatch logs and alarms. CPU: 256, Memory: 512MB, auto-scales 2-10.
  • 19.5.1.4 [High/M] Implement async job status API

    • File: apps/isis/generation-api/src/routes/jobs.ts ✓
    • Implement: GET /jobs/:id/status endpoint ✓
    • Implement: WebSocket for real-time updates (optional, deferred to future)
    • Acceptance: Clients can poll job status ✓
    • Note: Added JobStatusResponse type, getStatus service method, and route with cache control headers. Returns status, progress, queue position, timestamps, error (failed), workerId (running).

19.5.2 Result Handling#

  • 19.5.2.1 [High/M] Implement S3 presigned URL generation

    • File: libs/shared/storage/src/presigned.ts ✓
    • Implement: Generate presigned PUT URLs for RunPod uploads ✓
    • Config: 1-hour expiry, correct content-type ✓
    • Acceptance: RunPod can upload directly to S3 ✓
    • ✅ Completed: Constants (DEFAULT_UPLOAD_EXPIRY, MAX_EXPIRY), GPU_OUTPUT_CONTENT_TYPES mapping, detectContentType, generateGpuResultKey, validateExpiry, generateGpuUploadUrl, generateGpuDownloadUrl, generateBatchUploadUrls, createRunPodUploadInstructions. 28 tests.
  • 19.5.2.2 [High/M] Implement result callback handler

    • File: apps/infra/gpu-dispatcher/src/callbacks.ts ✓
    • Implement: Handle RunPod webhook callbacks ✓
    • Implement: Update job status, trigger downstream events ✓
    • Acceptance: Job completion updates propagate ✓
    • ✅ Completed: CallbackHandler HTTP server with webhook endpoint, HMAC signature validation, job mapping system, output extraction, downstream callback forwarding with signatures. 32 tests.
  • 19.5.2.3 [Medium/M] Implement result validation

    • File: libs/shared/gpu-dispatcher/src/validation.ts (new)
    • Implement: Verify uploaded files exist and valid
    • Implement: Check image dimensions, file integrity
    • Acceptance: Invalid results detected and flagged

19.5.3 Error Handling & Resilience#

  • 19.5.3.1 [High/M] Implement retry logic for RunPod failures

    • File: libs/shared/gpu-dispatcher/src/retry.ts (new)
    • Implement: Exponential backoff, max retries
    • Implement: Different retry strategies per error type
    • Acceptance: Transient failures automatically retried
    • ✅ Completed: Multiple strategies (exponential, linear, fibonacci, constant, aggressive, conservative), error categorization (transient, rate-limit, cold-start, resource, validation, timeout, cancelled), GPU-optimized configs, RetryPolicy class, 43 tests
  • 19.5.3.2 [High/M] Implement circuit breaker for RunPod

    • File: libs/shared/gpu-dispatcher/src/circuit-breaker.ts (new)
    • Implement: Open circuit on repeated failures
    • Implement: Half-open testing, automatic recovery
    • Acceptance: Failing endpoint doesn't cascade
  • 19.5.3.3 [High/S] Implement timeout handling

    • File: libs/shared/gpu-dispatcher/src/timeout.ts ✓
    • Implement: Job timeout (configurable per type) ✓
    • Implement: Cleanup on timeout ✓
    • Acceptance: Long-running jobs timeout gracefully ✓
  • 19.5.3.4 [Medium/M] Implement fallback endpoints

    • File: libs/shared/gpu-dispatcher/src/fallback.ts (new)
    • Implement: Try alternative endpoint/GPU if primary fails
    • Acceptance: Increased availability during partial outages

19.5.4 Cost Tracking#

  • 19.5.4.1 [Medium/M] Implement per-job cost tracking

    • File: libs/shared/gpu-dispatcher/src/cost-tracker.ts (new)
    • Implement: Track GPU-seconds per job
    • Implement: Calculate cost based on GPU type and duration
    • Acceptance: Cost per job recorded
  • 19.5.4.2 [Medium/M] Create cost dashboard metrics

    • File: libs/shared/gpu-dispatcher/src/metrics.ts (new)
    • Implement: Emit metrics for GPU usage, cost, queue depth
    • Acceptance: Metrics visible in CloudWatch
    • ✅ Completed: Prometheus-compatible metrics with job counters, queue gauges, duration histograms, cost tracking, endpoint health, circuit breaker state, fallback usage. GpuMetricsCollector class for dashboard snapshots. 37 tests added, total 205 tests.

19.6 Model Management#

19.6.1 Model Repository#

  • 19.6.1.1 [High/M] Create S3 bucket for model storage

    • File: infra/terraform/s3/models.tf ✓
    • Create: S3 bucket with versioning enabled ✓
    • Config: Lifecycle rules for old versions ✓
    • Acceptance: Bucket created, versioning works ✓
    • Note: Created complete Terraform module with versioning, 5 lifecycle rules (version transitions, checkpoints to Glacier, staging expiration, production to IA), SSE, public access block, intelligent tiering, optional cross-region replication, CloudWatch metrics. Includes dev/prod example tfvars.
  • 19.6.1.2 [High/M] Create model registry database schema

    • File: libs/isis/database/prisma/schema.prisma ✓ (in domain database)
    • Add: Model table (name, version, s3_path, size, type, metadata) ✓
    • Acceptance: Model metadata stored in database ✓
    • Note: Already implemented as ModelRegistry model in isis domain database (lines 566-628). Includes: name, version, description, type, format, storagePath, sizeBytes, hash, baseModel, trainedOn, triggerWords, tags, metadata, compatibleEngines, usage stats, ownership, status, timestamps.
  • 19.6.1.3 [High/M] Create model management API

    • File: apps/isis/generation-api/src/routes/models.ts ✓
    • Implement: List, upload, download, delete models ✓
    • Implement: Version management ✓
    • Acceptance: Models manageable via API ✓
    • Note: Created full REST API with POST /models (register), GET /models (list), GET /models/:id, PUT /models/:id (update), DELETE /models/:id (soft delete), GET /models/:id/versions, POST /models/:id/upload-url (presigned upload), POST /models/:id/upload-complete (finalize), GET /models/:id/download-url, GET /models/by-name/:name/:version. Includes model.service.ts (PostgreSQL), model.schema.ts (Zod validation), owner/org access control.

19.6.2 Model Caching Strategy#

  • 19.6.2.1 [High/M] Implement model baking in Docker images

    • Files: docker/runpod/*/Dockerfile ✓
    • Config: Common models baked into image layers ✓
    • Config: Use multi-stage builds to manage size ✓
    • Acceptance: Common models available instantly ✓
    • Note: Created multi-stage Docker builds for base and ComfyUI images. Base image includes CUDA 12.1.1, Python 3.10, core ML dependencies. ComfyUI image has 4 stages: base, models-sd15 (~4GB: SD 1.5, VAE, ControlNets, upscaler), models-sdxl (optional ~12GB), runtime. Includes model_utils.py for downloading, handler_base.py for RunPod integration.
  • 19.6.2.2 [Medium/M] Implement on-demand model download

    • Files: docker/runpod/*/handler.py ✓
    • Implement: Download models from S3 on cold start ✓
    • Implement: Cache in /runpod-volume if available ✓
    • Acceptance: Less common models loaded on demand ✓
    • Note: Created on_demand_loader.py with intelligent caching (baked → volume cache → S3 → HuggingFace → Registry). Handler extracts model names from job input and workflows, loads missing models before execution. Supports concurrent downloads, progress callbacks, automatic volume sync.
  • 19.6.2.3 Configure RunPod network volume (optional). Withdrawn 2026-09-18: network-volume configuration belongs in desired-state.json; RunPod endpoints are now declared in infra/runpod/endpoints/desired-state.json and reconciled by .github/workflows/runpod-endpoint-reconcile.yml (ISIS_CHROMA C.03, checked), not clicked in the RunPod console; this item is superseded by that mechanism.

19.6.3 Model Sync Pipeline#

  • 19.6.3.1 [Medium/M] Create model sync CLI tool

    • File: tools/model-sync/src/index.ts ✓
    • Implement: Sync models from S3 to RunPod volume ✓
    • Implement: Diff-based sync, progress reporting ✓
    • Acceptance: Models synced to RunPod storage ✓
    • Note: Created TypeScript CLI with sync, diff, list commands. Features: diff-based sync, concurrent downloads, progress bar with ETA, model type filtering, include/exclude patterns, checksum verification, dry-run mode, orphan detection, JSON output for scripting.
  • 19.6.3.2 [Medium/M] Create model update workflow

    • File: .github/workflows/model-sync.yml ✓
    • Config: Manual trigger to sync new models ✓
    • Acceptance: New models available on RunPod after sync ✓
    • Note: Created GitHub Actions workflow with analyze, sync, update-registry, notify jobs. Supports manual dispatch with model type filtering, include/exclude patterns, dry-run, force options. Scheduled weekly sync. Creates temporary RunPod pod for sync, auto-cleanup.

19.7 ComfyUI Migration (RunComfy → RunPod)#

19.7.1 Workflow Migration#

  • 19.7.1.1 Audit existing RunComfy workflows. Withdrawn 2026-09-18: the RunComfy migration was abandoned — RunComfy survives only as a provider enum in libs/yemaya/comfyui-integration, and ComfyUI-on-RunPod is owned by the ISIS_CHROMA track. AWS ECS/EKS were replaced by the single-box Hetzner compose plane (deploy-hetzner.yml), so this phase's premise is gone.

  • 19.7.1.2 Install required ComfyUI custom nodes. Withdrawn 2026-09-18: docker/runpod/comfyui/ flavor images are slated for retirement (ISIS_CHROMA C.12.05); the RunComfy migration is abandoned. AWS ECS/EKS were replaced by the single-box Hetzner compose plane (deploy-hetzner.yml), so this phase's premise is gone.

  • 19.7.1.3 Migrate workflow JSON files. Withdrawn 2026-09-18: assets/comfyui-workflows/ was never created and the RunComfy migration is abandoned; ComfyUI-on-RunPod is owned by ISIS_CHROMA.

  • 19.7.1.4 Create workflow validation tests. Withdrawn 2026-09-18: named a new tests/integration/comfyui-workflows.test.ts (which the conventions ratchet would reject as .test.ts) and needs a paid GPU endpoint; the RunComfy migration is abandoned and ComfyUI validation is owned by ISIS_CHROMA.

19.7.2 API Compatibility#

  • 19.7.2.1 [High/M] Create ComfyUI API adapter ✅

    • File: libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts (new)
    • Implemented: RunPodAdapter class with same interface as RunComfyProvider
    • Implemented: Job submission, polling, cancellation via RunPod serverless API
    • Implemented: Workflow translation from ComfyUI format to RunPod input
    • Implemented: Status mapping, output mapping, error handling
    • Implemented: Token bucket rate limiter, circuit breaker, retry policy
    • Implemented: Analytics, queue stats, retry stats tracking
    • Implemented: Factory functions (createRunPodAdapter, getRunPodAdapter)
    • Acceptance: Existing code works with minimal changes
  • 19.7.2.2 [High/M] Update Isis generation service ✅

    • File: apps/isis/generation-api/src/services/comfyui.service.ts (new)
    • Implemented: ComfyUIService with unified interface for ComfyUI execution
    • Implemented: Automatic provider selection based on feature flags
    • Implemented: Failover support between RunComfy and RunPod
    • Implemented: Health monitoring and provider health tracking
    • Implemented: Integration with workflow executor via initializeComfyUIService()
    • Implemented: Job statistics, analytics, and queue stats
    • Updated: WorkflowExecutor to use ComfyUIService for real execution
    • Updated: services/index.ts to export ComfyUIService
    • Acceptance: Generation requests route to RunPod
  • 19.7.2.3 [Medium/S] Add feature flag for migration ✅

    • File: libs/shared/config/src/features.ts (new)
    • Implemented: FeatureFlags registry with all migration flags
    • Implemented: COMFYUI_PROVIDER flag (runcomfy|runpod)
    • Implemented: ENABLE_RUNPOD_MIGRATION with percentage-based routing
    • Implemented: RUNPOD_*_ENDPOINT_ID configuration flags
    • Implemented: shouldUseRunPod(), getComfyUIProvider() helper functions
    • Implemented: validateRunPodConfiguration() for startup validation
    • Config: Image generation, performance, and experimental flags
    • Acceptance: Can switch between providers for testing

19.8 Observability & Monitoring#

19.8.1 CloudWatch Setup#

  • 19.8.1.1 [High/M] Create CloudWatch log groups ✅

    • File: infra/terraform/cloudwatch/logs.tf
    • Implemented: Log groups for all ECS services (flattened from domains)
    • Implemented: Infrastructure log groups (cluster, ALB access, VPC flow)
    • Implemented: RunPod metrics collector log group (conditional)
    • Implemented: Metric filters for errors, warnings, slow requests, HTTP status codes
    • Implemented: Business metric filters for Isis generation events
    • Implemented: Log Insights query definitions (error analysis, slow requests, etc.)
    • Config: 30-day retention, configurable via variables
    • Acceptance: All ECS logs in CloudWatch
  • 19.8.1.2 [High/M] Create CloudWatch dashboards ✅

    • File: infra/terraform/cloudwatch/dashboards.tf
    • Implemented: Platform overview dashboard (all services health)
    • Implemented: Per-domain dashboards with CPU, memory, error trends
    • Implemented: RunPod metrics dashboard with GPU utilization, queue depth, costs
    • Implemented: Isis generation business metrics dashboard
    • Implemented: Consistent color palette across dashboards
    • Acceptance: Dashboards show key metrics
  • 19.8.1.3 [High/M] Create CloudWatch alarms ✅

    • File: infra/terraform/cloudwatch/alarms.tf
    • Implemented: Cluster-level alarms (CPU, memory)
    • Implemented: Critical service alarms (CPU, memory, task count)
    • Implemented: Error rate and HTTP 5xx spike alarms per critical domain
    • Implemented: ALB latency (p99) and unhealthy hosts alarms
    • Implemented: RunPod alarms (hourly cost, queue depth, low utilization, failure rate)
    • Implemented: Composite alarm for critical services health
    • Config: SNS topic for notifications, configurable thresholds
    • Acceptance: Alarms trigger on threshold breach

19.8.2 RunPod Monitoring#

  • 19.8.2.1 [High/M] Create RunPod metrics collector ✅

    • File: apps/infra/runpod-metrics/src/index.ts (new)
    • Implemented: RunPodMetricsCollector class with health polling
    • Implemented: CloudWatch metric publishing (job counts, worker metrics)
    • Implemented: Calculated metrics (utilization, saturation, failure rate)
    • Implemented: Cost tracking metrics (estimated hourly cost)
    • Implemented: Aggregate metrics across all endpoints
    • Implemented: Configuration loading from environment variables
    • Implemented: Continuous and one-shot (Lambda) run modes
    • Config: tsup build, project.json, package.json
    • Acceptance: RunPod metrics in CloudWatch
  • 19.8.2.2 [Medium/M] Add RunPod metrics to dashboard ✅

    • File: infra/terraform/cloudwatch/dashboards.tf
    • Implemented: Dedicated RunPod dashboard with full metrics suite
    • Implemented: Workers running, jobs queued, overall utilization gauges
    • Implemented: Estimated hourly cost tracking
    • Implemented: Per-endpoint breakdown (workers, queue, utilization, failure rate)
    • Implemented: Jobs completed/failed over time
    • Implemented: Cost by endpoint with alert threshold annotations
    • Acceptance: RunPod metrics visible alongside AWS
  • 19.8.2.3 [Medium/M] Create RunPod cost alerts ✅

    • File: infra/terraform/cloudwatch/alarms.tf
    • Implemented: Hourly cost threshold alarm
    • Implemented: Queue depth threshold alarm
    • Implemented: Low utilization alarm (cost savings opportunity)
    • Implemented: High failure rate alarm (>10% across endpoints)
    • Config: Configurable thresholds via variables
    • Acceptance: Cost overruns detected quickly

19.8.3 Distributed Tracing#

  • 19.8.3.1 [Medium/L] Implement X-Ray tracing in ECS services ✅

    • File: libs/shared/tracing/src/xray.ts (new)
    • File: libs/shared/tracing/src/middleware/hono.ts (new)
    • Implemented: AWS X-Ray trace ID format utilities (parse, format, convert)
    • Implemented: X-Ray tracer configuration for ECS services
    • Implemented: ECS metadata fetching for trace enrichment
    • Implemented: X-Ray annotations and metadata utilities
    • Implemented: Hono tracing middleware (for Isis, Hathor, Yemaya services)
    • Updated: Existing Fastify/Express middleware already available
    • Updated: Package.json with X-Ray propagator dependency
    • Updated: pnpm catalog with @opentelemetry/propagator-aws-xray
    • Config: 5% sampling in production, 100% in development
    • Acceptance: Traces visible in X-Ray console
  • 19.8.3.2 [Medium/M] Implement trace propagation to RunPod ✅

    • File: libs/shared/gpu-dispatcher/src/tracing.ts (new)
    • Implemented: Trace context injection/extraction for GPU job metadata
    • Implemented: W3C traceparent header propagation to RunPod workers
    • Implemented: GpuJobTracer wrapper for traced dispatch, poll, complete, retry, upload
    • Implemented: Span attributes for GPU job lifecycle (type, status, priority, endpoint)
    • Implemented: Execution metrics recording in traces
    • Implemented: Job trace link creation for correlation
    • Updated: gpu-dispatcher index.ts with tracing exports
    • Updated: package.json with @oshun/tracing peer dependency
    • Acceptance: End-to-end traces include GPU execution

19.9 Deployment Pipeline#

19.9.1 CI/CD for ECS#

  • 19.9.1.1 [High/L] Create GitHub Actions workflow for ECS deploy ✅

    • File: .github/workflows/deploy-ecs.yml (new)
    • Implemented: Deployment planning with affected service detection
    • Implemented: Build and push to AWS ECR with matrix strategy
    • Implemented: ECS task definition update and service deployment
    • Implemented: Deployment stabilization wait and verification
    • Implemented: Post-deployment health checks
    • Implemented: Deployment summary with Slack notifications
    • Config: Per-environment deployment (staging via main, prod via tags/releases)
    • Config: Manual workflow dispatch with domain/service selection
    • Config: Force deploy option for manual runs
    • Acceptance: Commits to main deploy automatically
  • 19.9.1.2 [High/M] Implement blue/green deployment ✅

    • File: infra/terraform/ecs/deployment.tf (new)
    • Implemented: CodeDeploy application for ECS with blue/green deployment
    • Implemented: CodeDeploy deployment groups per service with traffic routing
    • Implemented: IAM role and policies for CodeDeploy
    • Implemented: Custom deployment configurations (Linear 5%, Canary 5%, Linear 25%)
    • Implemented: CloudWatch alarms for deployment health (error rate, latency, unhealthy hosts)
    • Implemented: Auto-rollback on deployment failure, stop, or alarm
    • Implemented: Optional approval workflow with configurable timeout
    • Implemented: SNS notifications for deployment events
    • Implemented: S3 bucket for AppSpec files (optional)
    • File: infra/terraform/ecs/variables.tf - Added blue/green variables
    • File: infra/terraform/ecs/outputs.tf - Added blue/green outputs
    • File: infra/terraform/ecs-services/blue-green.tf (new)
    • Implemented: Green target groups for blue/green services
    • Implemented: ECS services with CODE_DEPLOY controller
    • Implemented: Auto-scaling for blue/green services
    • File: infra/terraform/ecs-services/variables.tf - Added service-level blue/green options
    • File: infra/terraform/ecs-services/main.tf - Modified for rolling vs blue/green separation
    • File: infra/terraform/ecs-services/outputs.tf - Added blue/green target group outputs
    • Config: Automatic rollback on health check failure
    • Acceptance: Zero-downtime deployments
  • 19.9.1.3 [High/M] Create deployment approval workflow ✅

    • File: .github/workflows/deploy-ecs.yml
    • Implemented: Separate staging (deploy-staging) and production (deploy-production) jobs
    • Implemented: production-approval environment gate requiring manual approval
    • Implemented: approve-production job blocks until reviewer approval
    • Implemented: Staging deploys automatically (no approval needed)
    • Implemented: Production deploys blocked until approval via GitHub Environments
    • Implemented: Approval logging and summary reporting with timestamps
    • Implemented: More conservative deployment settings for production (maxPercent=150, failFast=true, maxParallel=2)
    • Implemented: Health check and summary jobs aware of both deployment paths
    • Config: Manual approval required for production via GitHub Environment protection rules
    • Note: Requires production-approval environment with Required Reviewers configured in GitHub repository Settings > Environments
    • Acceptance: Production deploys require human approval

19.9.2 CI/CD for RunPod#

  • 19.9.2.1 [High/M] Create GitHub Actions workflow for RunPod images ✅

    • File: .github/workflows/deploy-runpod.yml (new)
    • Implemented: Build GPU Docker images for RunPod serverless workers
    • Implemented: Automatic change detection via dorny/paths-filter
    • Implemented: Parallel builds for base, comfyui (sd15/sdxl), sd, flux, inference
    • Implemented: Base image built first, dependent images follow
    • Implemented: Docker Hub push on main branch merge
    • Implemented: Manual workflow dispatch with image selection
    • Implemented: GitHub Actions cache for faster rebuilds
    • Implemented: Multi-stage build support (ComfyUI sd15/sdxl variants)
    • Implemented: Build summary with status table
    • Implemented: Slack notifications on failure
    • Config: Separate from ECS pipeline
    • Config: Docker Hub namespace oshunai
    • Note: Requires DOCKERHUB_USERNAME and DOCKERHUB_TOKEN secrets
    • Acceptance: RunPod images built and pushed
  • 19.9.2.2 [Medium/M] Implement RunPod endpoint update automation ✅

    • File: .github/workflows/deploy-runpod.yml
    • Implemented: update-endpoints job using RunPod GraphQL API
    • Implemented: Automatic endpoint updates after successful image builds
    • Implemented: Endpoint ID mapping via GitHub repository variables
    • Implemented: Per-endpoint update status tracking
    • Implemented: Error handling with detailed logging
    • Implemented: Endpoint update summary in workflow output
    • Config: Endpoint IDs configured via GitHub vars (RUNPODENDPOINT*)
    • Note: Requires RUNPOD_API_KEY secret for authentication
    • Note: RunPod API doesn't support gradual rollout (instant updates)
    • Acceptance: Endpoints updated after image push

19.9.3 Infrastructure as Code#

  • 19.9.3.1 [High/M] Create Terraform state backend ✅

    • Directory: infra/terraform/state-backend/ (new module)
    • File: infra/terraform/state-backend/main.tf - S3 bucket, DynamoDB, IAM
    • File: infra/terraform/state-backend/variables.tf - Configuration options
    • File: infra/terraform/state-backend/outputs.tf - Backend config outputs
    • File: infra/terraform/state-backend/terraform.tfvars.example
    • Implemented: S3 bucket with versioning and server-side encryption
    • Implemented: DynamoDB table for state locking with PITR
    • Implemented: Optional KMS encryption for enhanced security
    • Implemented: IAM role for CI/CD with GitHub OIDC trust
    • Implemented: Bucket policy enforcing TLS and encryption
    • Implemented: Lifecycle rules for version management
    • Implemented: Auto-generated backend.tf template
    • Implemented: State migration script generator
    • Config: S3 bucket + DynamoDB for state locking
    • Acceptance: Terraform state stored remotely
  • 19.9.3.2 [High/M] Create Terraform CI workflow ✅

    • File: .github/workflows/terraform.yml (new)
    • Implemented: Change detection for Terraform modules using paths-filter
    • Implemented: Terraform format check (terraform fmt -check)
    • Implemented: Terraform validate for configuration syntax
    • Implemented: Plan jobs for staging and production environments
    • Implemented: Apply jobs with staging auto-apply on merge
    • Implemented: Production apply requires approval via GitHub Environment
    • Implemented: Plan output as PR comments for review
    • Implemented: Plan artifacts uploaded for apply job to use
    • Implemented: Workflow summary with module statuses
    • Implemented: Optional Slack notifications for apply results
    • Config: Plan on PR, apply on merge to main
    • Note: Requires production environment with Required Reviewers in GitHub
    • Acceptance: Infrastructure changes reviewed and applied
  • 19.9.3.3 [Medium/M] Create environment separation ✅

    • Directory: infra/terraform/environments/ (new)
    • File: infra/terraform/environments/README.md - Usage documentation
    • Files: infra/terraform/environments/staging/ - Staging environment
    • Files: infra/terraform/environments/production/ - Production environment
    • Implemented: Separate backend.tf with environment-specific state keys
    • Implemented: main.tf referencing all infrastructure modules
    • Implemented: variables.tf with environment-specific defaults
    • Implemented: outputs.tf for environment resources
    • Implemented: terraform.tfvars.example with environment values
    • Staging config: 2 AZs, single NAT, smaller instances, no deletion protection
    • Production config: 3 AZs, HA NAT, larger instances, deletion protection
    • Production config: Autoscaling enabled, KMS encryption, longer retention
    • Config: Separate VPC CIDRs to allow VPC peering if needed
    • Acceptance: Environments isolated

19.10 Decommissioning Old Infrastructure#

19.10.1 Traffic Migration#

  • 19.10.1.1 Configure DNS for gradual cutover (Route 53 weighted routing to ECS). Withdrawn 2026-09-18: ECS was retired; there is no ECS to cut traffic over to. Hetzner DNS is a single A record at the owner's registrar.

  • 19.10.1.2 Monitor error rates during cutover (CloudWatch). Withdrawn 2026-09-18: no ECS cutover happens; the Hetzner plane's monitoring is Prometheus/Grafana in infra/hetzner/observability.

  • 19.10.1.3 Complete DNS cutover (100% traffic to ECS). Withdrawn 2026-09-18: ECS was retired; there is no ECS target.

19.10.2 EKS Decommissioning#

  • 19.10.2.1 Drain EKS workloads. Withdrawn 2026-09-18: EKS was never the production plane and is retired; there is nothing to drain.

  • 19.10.2.2 Delete EKS cluster. Withdrawn 2026-09-18: no EKS cluster exists on the Hetzner plane.

  • 19.10.2.3 Clean up EKS IAM roles. Withdrawn 2026-09-18: no EKS/AWS IAM exists on the Hetzner plane.

19.10.3 RunComfy Decommissioning#

  • 19.10.3.1 Verify all workflows migrated. Withdrawn 2026-09-18: the RunComfy migration was abandoned; endpoint state is reconciled from desired-state.json (ISIS_CHROMA C.03).

  • 19.10.3.2 Cancel the RunComfy subscription to stop charges. An account action only the owner can take (stated 2026-09-18); RunComfy is no longer used (only a provider enum remains). blocked:external

  • 19.10.3.3 Export any remaining data stored on RunComfy before cancelling. Needs the owner's RunComfy account access to export (stated 2026-09-18). blocked:external

19.10.4 Cost Verification#

  • 19.10.4.1 Compare before/after costs (AWS and RunPod bills). Withdrawn 2026-09-18: there is no AWS/ECS bill — the Hetzner plane is a flat single-box cost; a Hetzner-vs-RunPod review is a different, owner-run finance task.

  • 19.10.4.2 Optimize infrastructure based on actual usage (right-size the Hetzner box and RunPod scaling). Needs real production traffic to right-size against (stated 2026-09-18). blocked:release


19.11 Documentation & Runbooks#

19.11.1 Architecture Documentation#

  • 19.11.1.1 [Medium/M] Document ECS architecture ✅

    • File: docs/infrastructure/ecs-architecture.md (new)
    • Content: Cluster setup, services, networking, scaling, monitoring, security
    • Content: Deployment strategies, health checks, troubleshooting
    • Acceptance: New team members can understand setup
  • 19.11.1.2 [Medium/M] Document RunPod integration ✅

    • File: docs/infrastructure/runpod-integration.md (new)
    • Content: Endpoints configuration, Docker images, handler code
    • Content: API integration, model management, monitoring, cost management
    • Acceptance: GPU workflow documented
  • 19.11.1.3 [Medium/M] Document deployment process ✅

    • File: docs/infrastructure/deployment.md (new)
    • Content: CI/CD pipelines, deployment stages, strategies
    • Content: Rollback procedures, health checks, best practices
    • Acceptance: Anyone can deploy

19.11.2 Operational Runbooks#

  • 19.11.2.1 [High/M] Create incident response runbook ✅

    • File: docs/reference/runbooks/incident-response.md (new)
    • Content: Severity levels, initial response checklist
    • Content: Common incidents diagnosis and resolution
    • Content: Communication templates, post-incident procedures
    • Acceptance: On-call can handle incidents
  • 19.11.2.2 [High/M] Create scaling runbook ✅

    • File: docs/reference/runbooks/scaling.md (new)
    • Content: ECS auto scaling, manual scaling, scheduled scaling
    • Content: RunPod workers scaling, database and cache scaling
    • Content: Emergency scaling procedures
    • Acceptance: Can handle traffic spikes
  • 19.11.2.3 [Medium/M] Create cost management runbook ✅

    • File: docs/reference/runbooks/cost-management.md (new)
    • Content: Cost monitoring with AWS Cost Explorer and budgets
    • Content: Optimization strategies for ECS, RDS, S3, RunPod
    • Content: Cost alerts, monthly reviews, emergency cost reduction
    • Acceptance: Can manage cloud costs

Phase 19 Summary Statistics#

Section Task Count
19.1 ECS Cluster Foundation 14
19.2 API Service Migration 24
19.3 Worker Migration (Non-GPU) 9
19.4 RunPod Serverless Setup 16
19.5 AWS ↔ RunPod Integration Layer 14
19.6 Model Management 8
19.7 ComfyUI Migration 7
19.8 Observability & Monitoring 9
19.9 Deployment Pipeline 9
19.10 Decommissioning Old Infrastructure 10
19.11 Documentation & Runbooks 6
TOTAL Phase 19 126

Updated Grand Total#

Phase Task Count
Phase 0-14 (Original) 796
Phase 15: Physical Wellness & Fitness 503
Phase 16: Video-to-Animation Pipeline 346
Phase 17: State-of-the-Art AI Integration 424
Phase 18: Production Readiness Remediation 847
Phase 19: AWS ECS + RunPod Infrastructure 126
GRAND TOTAL 3,042