# Oshun RunPod Integration This document describes how Oshun integrates with RunPod for GPU-accelerated AI workloads. ## Overview RunPod provides serverless GPU infrastructure for running AI inference and generation tasks. Oshun uses RunPod for: - Image generation (Stable Diffusion, SDXL, Flux) - Video generation - Audio processing - AI model inference ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Oshun Architecture │ │ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ AWS (ECS) │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ API │────▶│ Worker │────▶│ Queue │ │ │ │ │ │ Service │ │ Service │ │ (SQS) │ │ │ │ │ └─────────────┘ └──────┬──────┘ └─────────────┘ │ │ │ └─────────────────────────────│────────────────────────────────────────┘ │ │ │ │ │ │ HTTPS │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ RunPod │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌────────────┐ │ │ │ │ │ ComfyUI │ │ Flux │ │ SD │ │ Inference │ │ │ │ │ │ Endpoint │ │ Endpoint │ │ Endpoint │ │ Endpoint │ │ │ │ │ │ (A100/A40) │ │ (A100) │ │ (A40/4090) │ │ (Various) │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## Endpoints ### Endpoint Configuration Endpoints are configured via the RunPod dashboard and updated via the CI/CD pipeline. | Endpoint | GPU Type | Min Workers | Max Workers | Idle Timeout | | --------------------- | --------- | ----------- | ----------- | ------------ | | ComfyUI SD1.5 | A40 | 0 | 5 | 60s | | ComfyUI SDXL | A100 | 0 | 3 | 120s | | Character Consistency | A6000 | 0 | 2 | 60s | | Cinematic Video | A100 80GB | 0 | 2 | 90s | | Three-D | A100 80GB | 0 | 2 | 90s | | Texture | A6000 | 0 | 3 | 60s | | Flux | A100 | 0 | 3 | 120s | | SD (Basic) | 4090 | 0 | 10 | 30s | | Inference | Various | 0 | 5 | 60s | ### Endpoint Class Strategy (WS09-001) Isis classifies RunPod serverless lanes by three dimensions: - Asset complexity: `low` / `medium` / `high` / `extreme` - Control complexity: `basic` / `guided` / `consistency-heavy` / `orchestration-heavy` - Runtime profile: `interactive-short` / `interactive-medium` / `batch-long` / `sequence-long` | Class ID | Lifecycle | Primary Family | Asset Complexity | Control Complexity | Runtime Profile | Primary Workloads | | -------------------------- | --------- | -------------- | ---------------- | ------------------- | ------------------ | ------------------------------------- | | `comfyui-general` | active | `comfyui` | medium | guided | interactive-medium | general multi-modal generation | | `sd-fast-image` | active | `sd` | low | basic | interactive-short | fast image generation/editing | | `flux-high-fidelity-image` | active | `flux` | high | consistency-heavy | interactive-medium | high-fidelity image consistency | | `audio-sfx` | active | `audio-sfx` | medium | guided | batch-long | text-to-audio SFX | | `audio-voice` | active | `audio-voice` | medium | consistency-heavy | batch-long | voice synthesis | | `audio-music` | active | `audio-music` | high | orchestration-heavy | batch-long | music generation / multi-track cues | | `character-consistency` | planned | `comfyui` | high | consistency-heavy | interactive-medium | identity-retained character workflows | | `cinematic-video` | planned | `comfyui` | extreme | orchestration-heavy | sequence-long | shot/sequence video generation | | `three-d-asset` | planned | `comfyui` | extreme | orchestration-heavy | batch-long | 3D and geometry-heavy workflows | | `texture-optimization` | planned | `flux` | high | guided | interactive-medium | texture upscaling and material polish | ### Warm/Cold Policy Optimization (WS09-004) Desired-state worker warmness is now governed by workload class + SLA tier policy: - Policy source: `infra/runpod/endpoints/warm-cold-policy.json` - Policy applicator/checker: `scripts/isis/apply_runpod_warm_cold_policy.mjs` The policy maps each endpoint family to: - Workload class (`interactive-short`, `interactive-medium`, `batch-long`, `sequence-long`) - SLA tier (`critical`, `standard`, `economy`) - Warmness profile (`warm`, `balanced`, `cold`) Verification command: ```bash node scripts/isis/apply_runpod_warm_cold_policy.mjs --environment all --check ``` ### Pay-Per-Second Cost Attribution (WS09-005) RunPod runtime billing telemetry now includes deterministic attribution buckets for: - `workflow` - `project` - `tenant` - `endpoint` Attribution data is built from completed job runtime (`billableSeconds`), GPU tier pricing (USD/second), and metadata context propagated from submit routes (`workflowId`, `projectId`, `tenantId`/`organizationId`). `GET /api/v1/jobs/runpod/operations` now returns cost snapshots with: - `attribution.workflow.{credits,usd,seconds}` - `attribution.project.{credits,usd,seconds}` - `attribution.tenant.{credits,usd,seconds}` - `attribution.endpoint.{credits,usd,seconds}` Unscoped jobs are explicitly tracked in fallback buckets: - `__unscoped_project__` - `__unscoped_tenant__` - `__unknown_endpoint__` ### Per-Endpoint SLO Telemetry (WS09-006) RunPod operations telemetry now tracks endpoint-level SLO health using rolling runtime samples captured from completed, failed, and timeout outcomes. SLO objective defaults: - latency p95 `<= 180000ms` - success rate `>= 99%` - timeout rate `<= 1%` - queue wait p95 `<= 30000ms` - minimum sample size `5` before breach evaluation `GET /api/v1/jobs/runpod/operations` now includes `runpod.slos` with: - objective thresholds under `runpod.slos.objectives` - timestamped snapshot metadata (`generatedAt`) - endpoint entries containing: - `endpointId`, `family`, `channel`, `sampleSize` - `latencyP95Ms`, `successRatePct`, `timeoutRatePct`, `queueWaitP95Ms` - breach flags for each SLO dimension ### RunPod Anomaly Detection (WS09-007) RunPod operations now include explicit anomaly detection snapshots for: - cost spikes - timeout storms - model cache thrashing Detection runs over configurable sliding windows and emits active alerts when thresholds are breached. `GET /api/v1/jobs/runpod/operations` now returns `runpod.anomalies` with: - detection objectives/thresholds - cost-spike snapshot (`sampleSize`, `baseline`, `zScore`, `spike`) - timeout-storm entries per endpoint - model-cache-thrashing entries per endpoint - active alerts with type/severity/endpoint context ### Automated Canary Strategy and Rollback Triggers (WS09-008) RunPod operations now computes canary rollout strategy decisions per endpoint family using endpoint SLOs and anomaly telemetry. `GET /api/v1/jobs/runpod/operations` now includes `runpod.canaryStrategy` with: - current and target canary percentages per family - action (`promote`, `hold`, `rollback`, `steady`) - explicit rollback trigger flag - machine-readable reason list (SLO breach, endpoint anomaly alert, insufficient samples, global cost spike) Operational execution script: ```bash OPERATIONS_FILE=/tmp/runpod-operations.json \ FAIL_ON_ROLLBACK=true \ PROMOTE_CALLBACK='echo promote "$FAMILY" "$TARGET_PERCENTAGE"' \ ROLLBACK_CALLBACK='echo rollback "$FAMILY"' \ scripts/isis/execute_runpod_endpoint_canary_strategy.sh ``` The script executes promote/rollback callbacks and exits non-zero when rollback is triggered (unless `FAIL_ON_ROLLBACK=false`). ### Fleet Topology and Economics Console (WS09-009) RunPod operations now includes a fleet topology and economics snapshot that combines desired-state endpoint definitions with runtime attribution telemetry. `GET /api/v1/jobs/runpod/operations` now includes `runpod.fleet` with: - source metadata (`desiredStateFile`, load state, schema/update metadata) - topology inventory by environment and endpoint key: - endpoint family mapping - configured stable/canary endpoint IDs - worker min/max, GPU IDs, timeout envelopes - mapped endpoint class IDs - projected economics from configured workers and GPU hourly catalog: - totals (`$ / hour`, `$ / month`) - breakdowns by environment, family, and endpoint - observed economics rollups from runtime endpoint attribution: - by-endpoint USD/seconds - by-family USD/seconds The RunPod operator page now surfaces this topology/economics data as first-class tables so operators can inspect configured fleet shape and projected/observed cost posture in one place. ### Disaster Recovery Runbooks and Game-Day Exercises (WS09-010) RunPod disaster-recovery readiness now includes dedicated drills for both provider and endpoint-region failure classes: - Provider outage drill: - automation: `scripts/isis/run_provider_outage_drill.sh` - runbook: `docs/domains/isis/runbooks/provider-outage-drill.md` - Endpoint region failover drill: - automation: `scripts/isis/run_endpoint_region_failover_drill.sh` - runbook: `docs/domains/isis/runbooks/endpoint-region-failover-drill.md` - Composite game-day harness: - automation: `scripts/isis/run_runpod_disaster_recovery_gameday.sh` - runbook: `docs/domains/isis/runbooks/runpod-dr-gameday.md` The endpoint-region drill simulates regional loss by promoting a family canary endpoint to stable routing and validating failover plus restore invariants. ### Endpoint IDs Endpoint IDs are stored as GitHub repository variables: - `RUNPOD_ENDPOINT_COMFYUI_SD15` - ComfyUI SD1.5 endpoint - `RUNPOD_ENDPOINT_COMFYUI_SDXL` - ComfyUI SDXL endpoint - `RUNPOD_ENDPOINT_CHARACTER_CONSISTENCY` - Character consistency endpoint - `RUNPOD_ENDPOINT_CINEMATIC_VIDEO` - Cinematic video endpoint - `RUNPOD_ENDPOINT_THREE_D` - 3D endpoint - `RUNPOD_ENDPOINT_TEXTURE` - Texture endpoint - `RUNPOD_ENDPOINT_FLUX` - Flux endpoint - `RUNPOD_ENDPOINT_SD` - Stable Diffusion endpoint - `RUNPOD_ENDPOINT_INFERENCE` - General inference endpoint ## Docker Images ### Image Structure ``` docker/runpod/ ├── base/ # Base image with common dependencies │ ├── Dockerfile │ └── requirements.txt ├── comfyui/ # ComfyUI worker │ ├── Dockerfile.sd15 # SD1.5 variant │ ├── Dockerfile.sdxl # SDXL variant │ └── handler.py ├── sd/ # Stable Diffusion worker │ ├── Dockerfile │ └── handler.py ├── flux/ # Flux worker │ ├── Dockerfile │ └── handler.py └── inference/ # Generic inference worker ├── Dockerfile └── handler.py ``` ### Base Image The base image includes: - CUDA 12.1 runtime - Python 3.11 - PyTorch 2.1 - Common ML libraries (transformers, diffusers, accelerate) - RunPod SDK ### Building Images Images are built automatically by the CI/CD pipeline on changes to `docker/runpod/`, workflow runtime contracts, and endpoint manifest configuration. Manifest artifacts: - Build config source: `infra/runpod/endpoints/image-build-config.json` - Generated aggregate manifest: `infra/runpod/endpoints/image-build-manifest.json` - Generated per-endpoint inventories: `infra/runpod/endpoints/inventory/*.json` Generation and validation commands: ```bash node scripts/isis/generate_runpod_endpoint_image_manifests.mjs node scripts/isis/validate_runpod_endpoint_image_manifests.mjs ``` Manual build: ```bash # Build base image docker build -t oshunai/runpod-base:latest docker/runpod/base/ # Build ComfyUI image docker build -t oshunai/runpod-comfyui-sdxl:latest \ -f docker/runpod/comfyui/Dockerfile.sdxl \ docker/runpod/comfyui/ # Push to Docker Hub docker push oshunai/runpod-comfyui-sdxl:latest ``` ## Handler Code ### Handler Structure Each handler follows the RunPod serverless handler pattern: ```python import runpod def handler(event): """ Main handler function for RunPod serverless. Args: event: Dict containing: - input: The request payload - id: Unique job ID Returns: Dict with output data or error """ try: input_data = event["input"] # Validate input if "prompt" not in input_data: return {"error": "Missing required field: prompt"} # Process the request result = process_generation(input_data) return {"output": result} except Exception as e: return {"error": str(e)} runpod.serverless.start({"handler": handler}) ``` ### ComfyUI Handler The ComfyUI handler supports workflow-based generation: ```python def handler(event): input_data = event["input"] # Load workflow workflow = input_data.get("workflow") if workflow: # Use custom workflow workflow_data = json.loads(workflow) else: # Use default workflow based on type workflow_type = input_data.get("type", "txt2img") workflow_data = load_default_workflow(workflow_type) # Inject parameters into workflow inject_parameters(workflow_data, input_data) # Execute workflow result = execute_workflow(workflow_data) # Upload output to S3 output_url = upload_to_s3(result["images"]) return { "images": output_url, "seed": result["seed"], "execution_time": result["execution_time"] } ``` ### Input Schemas **Text-to-Image:** ```json { "prompt": "A beautiful sunset over mountains", "negative_prompt": "blurry, low quality", "width": 1024, "height": 1024, "steps": 30, "cfg_scale": 7.5, "sampler": "euler_ancestral", "seed": -1, "batch_size": 1 } ``` **Image-to-Image:** ```json { "prompt": "A beautiful sunset over mountains", "image": "https://example.com/input.png", "strength": 0.75, "width": 1024, "height": 1024, "steps": 30 } ``` **ComfyUI Workflow:** ```json { "workflow": "", "parameters": { "positive_prompt": "...", "negative_prompt": "...", "seed": 12345 } } ``` ## API Integration ### Making Requests From the Oshun worker service: ```typescript import { RunPodClient } from '@oshun/runpod-client'; const runpod = new RunPodClient({ apiKey: process.env.RUNPOD_API_KEY, endpoints: { comfyui: process.env.RUNPOD_ENDPOINT_COMFYUI_SDXL, flux: process.env.RUNPOD_ENDPOINT_FLUX, sd: process.env.RUNPOD_ENDPOINT_SD, }, }); // Synchronous request (wait for result) const result = await runpod.run('comfyui', { prompt: 'A beautiful sunset', width: 1024, height: 1024, }); // Asynchronous request (get job ID, poll for result) const job = await runpod.runAsync('comfyui', { prompt: 'A beautiful sunset', width: 1024, height: 1024, }); // Poll for result const result = await runpod.waitForJob(job.id, { timeout: 120000, pollInterval: 2000, }); ``` ### Webhook Integration For long-running jobs, use webhooks: ```typescript const job = await runpod.runAsync('flux', { prompt: 'A detailed landscape', webhook: 'https://api.oshun.ai/webhooks/runpod', webhook_events: ['completed', 'failed'], }); ``` Webhook payload: ```json { "id": "job-123", "status": "completed", "output": { "images": ["https://s3.../output.png"] }, "execution_time": 45.2 } ``` ## Model Management ### Model Storage Models are stored in RunPod network storage volumes: ``` /runpod-volume/ ├── models/ │ ├── checkpoints/ # SD checkpoints │ │ ├── sd15/ │ │ └── sdxl/ │ ├── loras/ # LoRA models │ ├── controlnet/ # ControlNet models │ └── embeddings/ # Textual inversions ├── outputs/ # Generated outputs (temporary) └── cache/ # Model cache ``` ### Adding New Models 1. Upload to network volume: ```bash runpodctl send models/checkpoints/sdxl/new_model.safetensors ``` 2. Update model registry in application: ```typescript // libs/isis/models/registry.ts export const MODEL_REGISTRY = { sdxl: { 'new-model': { path: '/runpod-volume/models/checkpoints/sdxl/new_model.safetensors', vae: 'sdxl-vae', type: 'checkpoint', }, }, }; ``` 3. Update handler to use new model: ```python # In handler.py MODEL_PATHS = { "new-model": "/runpod-volume/models/checkpoints/sdxl/new_model.safetensors", } ``` ### Model Preloading To reduce cold start times, models can be preloaded: ```python # In handler.py def preload_models(): """Preload frequently used models on worker startup.""" global PRELOADED_MODELS models_to_preload = ["sdxl-base", "sdxl-refiner"] for model_name in models_to_preload: model_path = MODEL_PATHS[model_name] PRELOADED_MODELS[model_name] = load_model(model_path) # Called on worker startup if __name__ == "__main__": preload_models() runpod.serverless.start({"handler": handler}) ``` ## Monitoring ### Endpoint Metrics Available via RunPod dashboard and API: - **Job metrics:** - Jobs completed - Jobs failed - Average execution time - Queue depth - **Worker metrics:** - Active workers - Idle workers - GPU utilization - Memory utilization ### CloudWatch Integration Metrics are pushed to CloudWatch for unified monitoring: ```typescript // In worker service async function trackRunPodMetrics(job: RunPodJob) { await cloudwatch.putMetricData({ Namespace: 'Oshun/RunPod', MetricData: [ { MetricName: 'JobExecutionTime', Value: job.executionTime, Unit: 'Seconds', Dimensions: [ { Name: 'Endpoint', Value: job.endpoint }, { Name: 'Environment', Value: process.env.ENVIRONMENT }, ], }, { MetricName: 'JobStatus', Value: job.status === 'completed' ? 1 : 0, Unit: 'Count', Dimensions: [ { Name: 'Endpoint', Value: job.endpoint }, { Name: 'Status', Value: job.status }, ], }, ], }); } ``` ### Alerts | Alert | Condition | Action | | -------------------- | -------------------- | ----------- | | High Error Rate | > 5% jobs failing | Slack + Ops | | Long Queue Time | > 2 min average wait | Slack | | Worker Scaling Issue | Max workers reached | Slack + Ops | | Cold Start Spikes | > 30s cold start avg | Slack | ## Cost Management ### Pricing Model RunPod charges per second of GPU time: | GPU Type | Price/Second | Price/Hour | | --------- | ------------ | ---------- | | RTX 4090 | $0.00014 | $0.50 | | A40 | $0.00024 | $0.86 | | A100 40GB | $0.00044 | $1.58 | | A100 80GB | $0.00056 | $2.00 | ### Cost Optimization 1. **Idle timeout configuration:** - Short timeout (30s) for infrequent endpoints - Longer timeout (120s) for high-traffic endpoints 2. **GPU selection:** - Use cheaper GPUs (4090, A40) for SD1.5 - Reserve A100s for SDXL/Flux 3. **Batching:** - Batch similar requests together - Use batch_size parameter when possible 4. **Model selection:** - Use quantized models (FP16) when quality permits - Consider smaller models for preview generation ### Cost Tracking ```typescript // In worker service async function trackCost(job: RunPodJob) { const gpuPricePerSecond = GPU_PRICING[job.gpuType]; const cost = job.executionTime * gpuPricePerSecond; await costTracker.record({ jobId: job.id, userId: job.userId, projectId: job.projectId, endpoint: job.endpoint, gpuType: job.gpuType, executionTime: job.executionTime, cost: cost, }); } ``` ## CI/CD Pipeline ### Workflow The GitHub Actions workflow (`.github/workflows/runpod-endpoint-image-build.yml`) handles: 1. **Manifest generation:** Build endpoint image and inventory manifests from desired state + workflow catalog. 2. **Warm/cold policy gate:** Validate desired-state worker sizing against workload-class + SLA-tier policy before any build starts. 3. **Manifest validation:** Enforce endpoint coverage, docker image parity, and per-endpoint inventory artifacts. 4. **Base image build:** Build/push `runpod-base` used by endpoint images. 5. **Endpoint matrix build:** Build/push endpoint-tagged images for all managed endpoint families. Manual lifecycle operations (create/update/delete/batch) are exposed via `.github/workflows/runpod-endpoint-lifecycle.yml`, which invokes `scripts/isis/runpod_endpoint_lifecycle.py` against the RunPod GraphQL API. Desired endpoint state is versioned in `infra/runpod/endpoints/desired-state.json` (GPU class, worker bounds, idle timeout, execution timeout, canary endpoint IDs, canary traffic percentages). Drift detection runs via `.github/workflows/runpod-endpoint-drift-detection.yml` using `scripts/isis/detect_runpod_endpoint_drift.py`. Drift reconciliation is exposed through `.github/workflows/runpod-endpoint-reconcile.yml` and `scripts/isis/reconcile_runpod_endpoint_drift.sh`. ### Canary Rollout by Endpoint RunPod staged rollout is controlled per endpoint type using endpoint-specific canary IDs and traffic percentages: - `RUNPOD_COMFYUI_CANARY_ENDPOINT_ID` + `RUNPOD_COMFYUI_CANARY_PERCENTAGE` - `RUNPOD_SD_CANARY_ENDPOINT_ID` + `RUNPOD_SD_CANARY_PERCENTAGE` - `RUNPOD_FLUX_CANARY_ENDPOINT_ID` + `RUNPOD_FLUX_CANARY_PERCENTAGE` - `RUNPOD_AUDIO_SFX_CANARY_ENDPOINT_ID` + `RUNPOD_AUDIO_SFX_CANARY_PERCENTAGE` - `RUNPOD_AUDIO_VOICE_CANARY_ENDPOINT_ID` + `RUNPOD_AUDIO_VOICE_CANARY_PERCENTAGE` - `RUNPOD_AUDIO_MUSIC_CANARY_ENDPOINT_ID` + `RUNPOD_AUDIO_MUSIC_CANARY_PERCENTAGE` The generation API routes requests deterministically (job/correlation/user metadata hash) so repeated retries for the same job stay on the same canary or stable endpoint during rollout. ### Pipeline Triggers - **Push to main:** Build endpoint image matrix and push tags - **Pull request:** Validate + build matrix without pushing - **Manual dispatch:** Build selected endpoint key subset, optionally push ### Updating Endpoints After image push, endpoints are updated via RunPod GraphQL API: ```python mutation = """ mutation UpdateEndpoint($input: UpdateEndpointInput!) { updateEndpoint(input: $input) { id name templateId } } """ variables = { "input": { "id": endpoint_id, "templateId": new_template_id, } } response = requests.post( "https://api.runpod.io/graphql", headers={"Authorization": f"Bearer {api_key}"}, json={"query": mutation, "variables": variables} ) ``` ### Lifecycle Automation Script The lifecycle script supports `create`, `update`, `delete`, and `batch` operations, plus `apply-desired-state`: ```bash # Dry-run update python scripts/isis/runpod_endpoint_lifecycle.py \ --operation update \ --endpoint-spec-file /tmp/endpoint.json \ --dry-run # Batch create/update/delete python scripts/isis/runpod_endpoint_lifecycle.py \ --operation batch \ --operations-file /tmp/operations.json # Apply desired-state for production endpoints python scripts/isis/runpod_endpoint_lifecycle.py \ --operation apply-desired-state \ --desired-state-file infra/runpod/endpoints/desired-state.json \ --environment production # Detect + reconcile drift end-to-end bash scripts/isis/reconcile_runpod_endpoint_drift.sh ``` ## Troubleshooting ### Common Issues **Job stuck in queue:** ```bash # Check endpoint status curl -X POST https://api.runpod.io/graphql \ -H "Authorization: Bearer $RUNPOD_API_KEY" \ -d '{"query": "{ myself { endpoints { id name workersMax workersMin } } }"}' # Possible causes: # - No workers available (increase max workers) # - Workers failing to start (check image/volume) # - GPU type unavailable (check region) ``` **Worker crashes:** 1. Check worker logs in RunPod dashboard 2. Common causes: - OOM (increase GPU memory or reduce batch size) - Model loading failure (check volume mount) - Handler exception (check handler code) **Slow cold starts:** - Optimize Docker image (smaller layers) - Preload critical models - Increase idle timeout - Consider reserved workers **Image pull failures:** ```bash # Verify image exists docker pull oshunai/runpod-comfyui-sdxl:latest # Check Docker Hub credentials in RunPod # Template > Docker Credentials ``` ### Useful Commands ```bash # Get job status curl https://api.runpod.ai/v2/$ENDPOINT_ID/status/$JOB_ID \ -H "Authorization: Bearer $RUNPOD_API_KEY" # Cancel job curl -X POST https://api.runpod.ai/v2/$ENDPOINT_ID/cancel/$JOB_ID \ -H "Authorization: Bearer $RUNPOD_API_KEY" # Purge queue curl -X POST https://api.runpod.ai/v2/$ENDPOINT_ID/purge-queue \ -H "Authorization: Bearer $RUNPOD_API_KEY" ``` ## Related Documentation - [ECS Architecture](./ecs-architecture.md) - [Deployment Process](./deployment.md) - [Cost Management Runbook](../reference/runbooks/cost-management.md) - Isis RunPod on-call runbook (planned — not yet written) - Isis provider-outage drill runbook (planned — not yet written) - Isis endpoint region-failover drill runbook (planned — not yet written)