Oshun RunPod Integration

This document describes how Oshun integrates with RunPod for GPU-accelerated AI workloads.

Overview#

RunPod provides serverless GPU infrastructure for running AI inference and generation tasks. Oshun uses RunPod for:

• Image generation (Stable Diffusion, SDXL, Flux)
• Video generation
• Audio processing
• AI model inference

┌─────────────────────────────────────────────────────────────────────────────┐
│                           Oshun Architecture                                 │
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                         AWS (ECS)                                    │    │
│  │  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐           │    │
│  │  │   API       │────▶│   Worker    │────▶│   Queue     │           │    │
│  │  │   Service   │     │   Service   │     │   (SQS)     │           │    │
│  │  └─────────────┘     └──────┬──────┘     └─────────────┘           │    │
│  └─────────────────────────────│────────────────────────────────────────┘    │
│                                │                                             │
│                                │ HTTPS                                       │
│                                ▼                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                         RunPod                                       │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌────────────┐ │    │
│  │  │  ComfyUI    │  │   Flux      │  │   SD        │  │ Inference  │ │    │
│  │  │  Endpoint   │  │  Endpoint   │  │  Endpoint   │  │  Endpoint  │ │    │
│  │  │  (A100/A40) │  │   (A100)    │  │  (A40/4090) │  │  (Various) │ │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └────────────┘ │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Endpoints#

Endpoint Configuration#

Endpoints are configured via the RunPod dashboard and updated via the CI/CD pipeline.

Endpoint Class Strategy (WS09-001)#

Isis classifies RunPod serverless lanes by three dimensions:

• Asset complexity: low / medium / high / extreme
• Control complexity: basic / guided / consistency-heavy / orchestration-heavy
• Runtime profile: interactive-short / interactive-medium / batch-long / sequence-long

Warm/Cold Policy Optimization (WS09-004)#

Desired-state worker warmness is now governed by workload class + SLA tier policy:

• Policy source: infra/runpod/endpoints/warm-cold-policy.json
• Policy applicator/checker: scripts/isis/apply_runpod_warm_cold_policy.mjs

The policy maps each endpoint family to:

• Workload class (interactive-short, interactive-medium, batch-long, sequence-long)
• SLA tier (critical, standard, economy)
• Warmness profile (warm, balanced, cold)

Verification command:

node scripts/isis/apply_runpod_warm_cold_policy.mjs --environment all --check

Pay-Per-Second Cost Attribution (WS09-005)#

RunPod runtime billing telemetry now includes deterministic attribution buckets for:

• workflow
• project
• tenant
• endpoint

Attribution data is built from completed job runtime (billableSeconds), GPU tier pricing (USD/second), and metadata context propagated from submit routes (workflowId, projectId, tenantId/organizationId).

GET /api/v1/jobs/runpod/operations now returns cost snapshots with:

• attribution.workflow.{credits,usd,seconds}
• attribution.project.{credits,usd,seconds}
• attribution.tenant.{credits,usd,seconds}
• attribution.endpoint.{credits,usd,seconds}

Unscoped jobs are explicitly tracked in fallback buckets:

• __unscoped_project__
• __unscoped_tenant__
• __unknown_endpoint__

Per-Endpoint SLO Telemetry (WS09-006)#

RunPod operations telemetry now tracks endpoint-level SLO health using rolling runtime samples captured from completed, failed, and timeout outcomes.

SLO objective defaults:

• latency p95 <= 180000ms
• success rate >= 99%
• timeout rate <= 1%
• queue wait p95 <= 30000ms
• minimum sample size 5 before breach evaluation

GET /api/v1/jobs/runpod/operations now includes runpod.slos with:

• objective thresholds under runpod.slos.objectives
• timestamped snapshot metadata (generatedAt)
• endpoint entries containing:
  • endpointId, family, channel, sampleSize
  • latencyP95Ms, successRatePct, timeoutRatePct, queueWaitP95Ms
  • breach flags for each SLO dimension

RunPod Anomaly Detection (WS09-007)#

RunPod operations now include explicit anomaly detection snapshots for:

• cost spikes
• timeout storms
• model cache thrashing

Detection runs over configurable sliding windows and emits active alerts when thresholds are breached.

GET /api/v1/jobs/runpod/operations now returns runpod.anomalies with:

• detection objectives/thresholds
• cost-spike snapshot (sampleSize, baseline, zScore, spike)
• timeout-storm entries per endpoint
• model-cache-thrashing entries per endpoint
• active alerts with type/severity/endpoint context

Automated Canary Strategy and Rollback Triggers (WS09-008)#

RunPod operations now computes canary rollout strategy decisions per endpoint family using endpoint SLOs and anomaly telemetry.

GET /api/v1/jobs/runpod/operations now includes runpod.canaryStrategy with:

• current and target canary percentages per family
• action (promote, hold, rollback, steady)
• explicit rollback trigger flag
• machine-readable reason list (SLO breach, endpoint anomaly alert, insufficient samples, global cost spike)

Operational execution script:

OPERATIONS_FILE=/tmp/runpod-operations.json \
FAIL_ON_ROLLBACK=true \
PROMOTE_CALLBACK='echo promote "$FAMILY" "$TARGET_PERCENTAGE"' \
ROLLBACK_CALLBACK='echo rollback "$FAMILY"' \
scripts/isis/execute_runpod_endpoint_canary_strategy.sh

The script executes promote/rollback callbacks and exits non-zero when rollback is triggered (unless FAIL_ON_ROLLBACK=false).

Fleet Topology and Economics Console (WS09-009)#

RunPod operations now includes a fleet topology and economics snapshot that combines desired-state endpoint definitions with runtime attribution telemetry.

GET /api/v1/jobs/runpod/operations now includes runpod.fleet with:

• source metadata (desiredStateFile, load state, schema/update metadata)
• topology inventory by environment and endpoint key:
  • endpoint family mapping
  • configured stable/canary endpoint IDs
  • worker min/max, GPU IDs, timeout envelopes
  • mapped endpoint class IDs
• projected economics from configured workers and GPU hourly catalog:
  • totals ($ / hour, $ / month)
  • breakdowns by environment, family, and endpoint
• observed economics rollups from runtime endpoint attribution:
  • by-endpoint USD/seconds
  • by-family USD/seconds

The RunPod operator page now surfaces this topology/economics data as first-class tables so operators can inspect configured fleet shape and projected/observed cost posture in one place.

Disaster Recovery Runbooks and Game-Day Exercises (WS09-010)#

RunPod disaster-recovery readiness now includes dedicated drills for both provider and endpoint-region failure classes:

• Provider outage drill:
  • automation: scripts/isis/run_provider_outage_drill.sh
  • runbook: docs/domains/isis/runbooks/provider-outage-drill.md
• Endpoint region failover drill:
  • automation: scripts/isis/run_endpoint_region_failover_drill.sh
  • runbook: docs/domains/isis/runbooks/endpoint-region-failover-drill.md
• Composite game-day harness:
  • automation: scripts/isis/run_runpod_disaster_recovery_gameday.sh
  • runbook: docs/domains/isis/runbooks/runpod-dr-gameday.md

The endpoint-region drill simulates regional loss by promoting a family canary endpoint to stable routing and validating failover plus restore invariants.

Endpoint IDs#

Endpoint IDs are stored as GitHub repository variables:

• RUNPOD_ENDPOINT_COMFYUI_SD15 - ComfyUI SD1.5 endpoint
• RUNPOD_ENDPOINT_COMFYUI_SDXL - ComfyUI SDXL endpoint
• RUNPOD_ENDPOINT_CHARACTER_CONSISTENCY - Character consistency endpoint
• RUNPOD_ENDPOINT_CINEMATIC_VIDEO - Cinematic video endpoint
• RUNPOD_ENDPOINT_THREE_D - 3D endpoint
• RUNPOD_ENDPOINT_TEXTURE - Texture endpoint
• RUNPOD_ENDPOINT_FLUX - Flux endpoint
• RUNPOD_ENDPOINT_SD - Stable Diffusion endpoint
• RUNPOD_ENDPOINT_INFERENCE - General inference endpoint

Docker Images#

Image Structure#

docker/runpod/
├── base/                    # Base image with common dependencies
│   ├── Dockerfile
│   └── requirements.txt
├── comfyui/                 # ComfyUI worker
│   ├── Dockerfile.sd15      # SD1.5 variant
│   ├── Dockerfile.sdxl      # SDXL variant
│   └── handler.py
├── sd/                      # Stable Diffusion worker
│   ├── Dockerfile
│   └── handler.py
├── flux/                    # Flux worker
│   ├── Dockerfile
│   └── handler.py
└── inference/               # Generic inference worker
    ├── Dockerfile
    └── handler.py

Base Image#

The base image includes:

• CUDA 12.1 runtime
• Python 3.11
• PyTorch 2.1
• Common ML libraries (transformers, diffusers, accelerate)
• RunPod SDK

Building Images#

Images are built automatically by the CI/CD pipeline on changes to docker/runpod/, workflow runtime contracts, and endpoint manifest configuration.

Manifest artifacts:

• Build config source: infra/runpod/endpoints/image-build-config.json
• Generated aggregate manifest: infra/runpod/endpoints/image-build-manifest.json
• Generated per-endpoint inventories: infra/runpod/endpoints/inventory/*.json

Generation and validation commands:

node scripts/isis/generate_runpod_endpoint_image_manifests.mjs
node scripts/isis/validate_runpod_endpoint_image_manifests.mjs

Manual build:

# Build base image
docker build -t oshunai/runpod-base:latest docker/runpod/base/

# Build ComfyUI image
docker build -t oshunai/runpod-comfyui-sdxl:latest \
  -f docker/runpod/comfyui/Dockerfile.sdxl \
  docker/runpod/comfyui/

# Push to Docker Hub
docker push oshunai/runpod-comfyui-sdxl:latest

Handler Code#

Handler Structure#

Each handler follows the RunPod serverless handler pattern:

import runpod

def handler(event):
    """
    Main handler function for RunPod serverless.

    Args:
        event: Dict containing:
            - input: The request payload
            - id: Unique job ID

    Returns:
        Dict with output data or error
    """
    try:
        input_data = event["input"]

        # Validate input
        if "prompt" not in input_data:
            return {"error": "Missing required field: prompt"}

        # Process the request
        result = process_generation(input_data)

        return {"output": result}

    except Exception as e:
        return {"error": str(e)}


runpod.serverless.start({"handler": handler})

ComfyUI Handler#

The ComfyUI handler supports workflow-based generation:

def handler(event):
    input_data = event["input"]

    # Load workflow
    workflow = input_data.get("workflow")
    if workflow:
        # Use custom workflow
        workflow_data = json.loads(workflow)
    else:
        # Use default workflow based on type
        workflow_type = input_data.get("type", "txt2img")
        workflow_data = load_default_workflow(workflow_type)

    # Inject parameters into workflow
    inject_parameters(workflow_data, input_data)

    # Execute workflow
    result = execute_workflow(workflow_data)

    # Upload output to S3
    output_url = upload_to_s3(result["images"])

    return {
        "images": output_url,
        "seed": result["seed"],
        "execution_time": result["execution_time"]
    }

Input Schemas#

Text-to-Image:

{
  "prompt": "A beautiful sunset over mountains",
  "negative_prompt": "blurry, low quality",
  "width": 1024,
  "height": 1024,
  "steps": 30,
  "cfg_scale": 7.5,
  "sampler": "euler_ancestral",
  "seed": -1,
  "batch_size": 1
}

Image-to-Image:

{
  "prompt": "A beautiful sunset over mountains",
  "image": "https://example.com/input.png",
  "strength": 0.75,
  "width": 1024,
  "height": 1024,
  "steps": 30
}

ComfyUI Workflow:

{
  "workflow": "<base64 encoded workflow JSON>",
  "parameters": {
    "positive_prompt": "...",
    "negative_prompt": "...",
    "seed": 12345
  }
}

API Integration#

Making Requests#

From the Oshun worker service:

import { RunPodClient } from '@oshun/runpod-client';

const runpod = new RunPodClient({
  apiKey: process.env.RUNPOD_API_KEY,
  endpoints: {
    comfyui: process.env.RUNPOD_ENDPOINT_COMFYUI_SDXL,
    flux: process.env.RUNPOD_ENDPOINT_FLUX,
    sd: process.env.RUNPOD_ENDPOINT_SD,
  },
});

// Synchronous request (wait for result)
const result = await runpod.run('comfyui', {
  prompt: 'A beautiful sunset',
  width: 1024,
  height: 1024,
});

// Asynchronous request (get job ID, poll for result)
const job = await runpod.runAsync('comfyui', {
  prompt: 'A beautiful sunset',
  width: 1024,
  height: 1024,
});

// Poll for result
const result = await runpod.waitForJob(job.id, {
  timeout: 120000,
  pollInterval: 2000,
});

Webhook Integration#

For long-running jobs, use webhooks:

const job = await runpod.runAsync('flux', {
  prompt: 'A detailed landscape',
  webhook: 'https://api.oshun.ai/webhooks/runpod',
  webhook_events: ['completed', 'failed'],
});

Webhook payload:

{
  "id": "job-123",
  "status": "completed",
  "output": {
    "images": ["https://s3.../output.png"]
  },
  "execution_time": 45.2
}

Model Management#

Model Storage#

Models are stored in RunPod network storage volumes:

/runpod-volume/
├── models/
│   ├── checkpoints/         # SD checkpoints
│   │   ├── sd15/
│   │   └── sdxl/
│   ├── loras/               # LoRA models
│   ├── controlnet/          # ControlNet models
│   └── embeddings/          # Textual inversions
├── outputs/                  # Generated outputs (temporary)
└── cache/                    # Model cache

Adding New Models#

1. Upload to network volume:

runpodctl send models/checkpoints/sdxl/new_model.safetensors

2. Update model registry in application:

// libs/isis/models/registry.ts
export const MODEL_REGISTRY = {
  sdxl: {
    'new-model': {
      path: '/runpod-volume/models/checkpoints/sdxl/new_model.safetensors',
      vae: 'sdxl-vae',
      type: 'checkpoint',
    },
  },
};

3. Update handler to use new model:

# In handler.py
MODEL_PATHS = {
    "new-model": "/runpod-volume/models/checkpoints/sdxl/new_model.safetensors",
}

Model Preloading#

To reduce cold start times, models can be preloaded:

# In handler.py
def preload_models():
    """Preload frequently used models on worker startup."""
    global PRELOADED_MODELS

    models_to_preload = ["sdxl-base", "sdxl-refiner"]

    for model_name in models_to_preload:
        model_path = MODEL_PATHS[model_name]
        PRELOADED_MODELS[model_name] = load_model(model_path)

# Called on worker startup
if __name__ == "__main__":
    preload_models()
    runpod.serverless.start({"handler": handler})

Monitoring#

Endpoint Metrics#

Available via RunPod dashboard and API:

• Job metrics:

  • Jobs completed
  • Jobs failed
  • Average execution time
  • Queue depth

• Worker metrics:

  • Active workers
  • Idle workers
  • GPU utilization
  • Memory utilization

CloudWatch Integration#

Metrics are pushed to CloudWatch for unified monitoring:

// In worker service
async function trackRunPodMetrics(job: RunPodJob) {
  await cloudwatch.putMetricData({
    Namespace: 'Oshun/RunPod',
    MetricData: [
      {
        MetricName: 'JobExecutionTime',
        Value: job.executionTime,
        Unit: 'Seconds',
        Dimensions: [
          { Name: 'Endpoint', Value: job.endpoint },
          { Name: 'Environment', Value: process.env.ENVIRONMENT },
        ],
      },
      {
        MetricName: 'JobStatus',
        Value: job.status === 'completed' ? 1 : 0,
        Unit: 'Count',
        Dimensions: [
          { Name: 'Endpoint', Value: job.endpoint },
          { Name: 'Status', Value: job.status },
        ],
      },
    ],
  });
}

Alerts#

Cost Management#

Pricing Model#

RunPod charges per second of GPU time:

Cost Optimization#

1. Idle timeout configuration:

   • Short timeout (30s) for infrequent endpoints
   • Longer timeout (120s) for high-traffic endpoints

2. GPU selection:

   • Use cheaper GPUs (4090, A40) for SD1.5
   • Reserve A100s for SDXL/Flux

3. Batching:

   • Batch similar requests together
   • Use batch_size parameter when possible

4. Model selection:

   • Use quantized models (FP16) when quality permits
   • Consider smaller models for preview generation

Cost Tracking#

// In worker service
async function trackCost(job: RunPodJob) {
  const gpuPricePerSecond = GPU_PRICING[job.gpuType];
  const cost = job.executionTime * gpuPricePerSecond;

  await costTracker.record({
    jobId: job.id,
    userId: job.userId,
    projectId: job.projectId,
    endpoint: job.endpoint,
    gpuType: job.gpuType,
    executionTime: job.executionTime,
    cost: cost,
  });
}

CI/CD Pipeline#

Workflow#

The GitHub Actions workflow (.github/workflows/runpod-endpoint-image-build.yml) handles:

1. Manifest generation: Build endpoint image and inventory manifests from desired state + workflow catalog.
2. Warm/cold policy gate: Validate desired-state worker sizing against workload-class + SLA-tier policy before any build starts.
3. Manifest validation: Enforce endpoint coverage, docker image parity, and per-endpoint inventory artifacts.
4. Base image build: Build/push runpod-base used by endpoint images.
5. Endpoint matrix build: Build/push endpoint-tagged images for all managed endpoint families.

Manual lifecycle operations (create/update/delete/batch) are exposed via .github/workflows/runpod-endpoint-lifecycle.yml, which invokes scripts/isis/runpod_endpoint_lifecycle.py against the RunPod GraphQL API. Desired endpoint state is versioned in infra/runpod/endpoints/desired-state.json (GPU class, worker bounds, idle timeout, execution timeout, canary endpoint IDs, canary traffic percentages). Drift detection runs via .github/workflows/runpod-endpoint-drift-detection.yml using scripts/isis/detect_runpod_endpoint_drift.py. Drift reconciliation is exposed through .github/workflows/runpod-endpoint-reconcile.yml and scripts/isis/reconcile_runpod_endpoint_drift.sh.

Canary Rollout by Endpoint#

RunPod staged rollout is controlled per endpoint type using endpoint-specific canary IDs and traffic percentages:

• RUNPOD_COMFYUI_CANARY_ENDPOINT_ID + RUNPOD_COMFYUI_CANARY_PERCENTAGE
• RUNPOD_SD_CANARY_ENDPOINT_ID + RUNPOD_SD_CANARY_PERCENTAGE
• RUNPOD_FLUX_CANARY_ENDPOINT_ID + RUNPOD_FLUX_CANARY_PERCENTAGE
• RUNPOD_AUDIO_SFX_CANARY_ENDPOINT_ID + RUNPOD_AUDIO_SFX_CANARY_PERCENTAGE
• RUNPOD_AUDIO_VOICE_CANARY_ENDPOINT_ID + RUNPOD_AUDIO_VOICE_CANARY_PERCENTAGE
• RUNPOD_AUDIO_MUSIC_CANARY_ENDPOINT_ID + RUNPOD_AUDIO_MUSIC_CANARY_PERCENTAGE

The generation API routes requests deterministically (job/correlation/user metadata hash) so repeated retries for the same job stay on the same canary or stable endpoint during rollout.

Pipeline Triggers#

• Push to main: Build endpoint image matrix and push tags
• Pull request: Validate + build matrix without pushing
• Manual dispatch: Build selected endpoint key subset, optionally push

Updating Endpoints#

After image push, endpoints are updated via RunPod GraphQL API:

mutation = """
mutation UpdateEndpoint($input: UpdateEndpointInput!) {
    updateEndpoint(input: $input) {
        id
        name
        templateId
    }
}
"""

variables = {
    "input": {
        "id": endpoint_id,
        "templateId": new_template_id,
    }
}

response = requests.post(
    "https://api.runpod.io/graphql",
    headers={"Authorization": f"Bearer {api_key}"},
    json={"query": mutation, "variables": variables}
)

Lifecycle Automation Script#

The lifecycle script supports create, update, delete, and batch operations, plus apply-desired-state:

# Dry-run update
python scripts/isis/runpod_endpoint_lifecycle.py \
  --operation update \
  --endpoint-spec-file /tmp/endpoint.json \
  --dry-run

# Batch create/update/delete
python scripts/isis/runpod_endpoint_lifecycle.py \
  --operation batch \
  --operations-file /tmp/operations.json

# Apply desired-state for production endpoints
python scripts/isis/runpod_endpoint_lifecycle.py \
  --operation apply-desired-state \
  --desired-state-file infra/runpod/endpoints/desired-state.json \
  --environment production

# Detect + reconcile drift end-to-end
bash scripts/isis/reconcile_runpod_endpoint_drift.sh

Troubleshooting#

Common Issues#

Job stuck in queue:

# Check endpoint status
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { endpoints { id name workersMax workersMin } } }"}'

# Possible causes:
# - No workers available (increase max workers)
# - Workers failing to start (check image/volume)
# - GPU type unavailable (check region)

Worker crashes:

1. Check worker logs in RunPod dashboard
2. Common causes:
   • OOM (increase GPU memory or reduce batch size)
   • Model loading failure (check volume mount)
   • Handler exception (check handler code)

Slow cold starts:

• Optimize Docker image (smaller layers)
• Preload critical models
• Increase idle timeout
• Consider reserved workers

Image pull failures:

# Verify image exists
docker pull oshunai/runpod-comfyui-sdxl:latest

# Check Docker Hub credentials in RunPod
# Template > Docker Credentials

Useful Commands#

# Get job status
curl https://api.runpod.ai/v2/$ENDPOINT_ID/status/$JOB_ID \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

# Cancel job
curl -X POST https://api.runpod.ai/v2/$ENDPOINT_ID/cancel/$JOB_ID \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

# Purge queue
curl -X POST https://api.runpod.ai/v2/$ENDPOINT_ID/purge-queue \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

Related Documentation#

• ECS Architecture
• Deployment Process
• Cost Management Runbook
• Isis RunPod on-call runbook (planned — not yet written)
• Isis provider-outage drill runbook (planned — not yet written)
• Isis endpoint region-failover drill runbook (planned — not yet written)