Disciplines · Integrations

ComfyUI Integration Documentation

ComfyUI is a node-based interface for Stable Diffusion and other image generation models.

12sections2 minread
Markdown source
On this page

This directory contains comprehensive documentation for integrating ComfyUI image generation capabilities into the Oshun platform.

Document Description
Workflow Documentation Creating and executing workflows
Node Reference Common nodes and their configurations
Performance Tuning Optimization and scaling strategies
Custom Node Development Building custom nodes
RunComfy API Client API client reference

Overview#

ComfyUI is a node-based interface for Stable Diffusion and other image generation models. Oshun integrates with ComfyUI through multiple providers for scalable, production-grade image generation.

Key Features#

  • Multi-Provider Support: RunComfy, RunPod, Modal, Replicate, Vast.ai
  • Intelligent Routing: Cost-optimized, latency-optimized, or reliability modes
  • Automatic Failover: Seamless switching between providers
  • Workflow Flexibility: Built-in types + custom workflow support
  • Production Ready: Rate limiting, circuit breakers, comprehensive monitoring

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                        Oshun ComfyUI Integration                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Application Layer                                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  Lilith Consciousness  │  Yemaya Creative    │    Isis Factory      │    │
│  │  - Avatar generation   │  - Asset creation   │    - Batch gen       │    │
│  │  - Real-time preview   │  - Style transfer   │    - Upscaling       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                      │                                       │
│  Service Layer                       ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                     ComfyUI Service                                  │    │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │    │
│  │  │ RunComfy     │  │ RunPod       │  │ Comfy Cloud  │              │    │
│  │  │ Provider     │  │ Adapter      │  │ Provider     │              │    │
│  │  └──────────────┘  └──────────────┘  └──────────────┘              │    │
│  │                                                                      │    │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │    │
│  │  │ Rate Limiter │  │ Circuit      │  │ Retry        │              │    │
│  │  │              │  │ Breaker      │  │ Handler      │              │    │
│  │  └──────────────┘  └──────────────┘  └──────────────┘              │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                      │                                       │
│  Provider Layer                      ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  RunComfy    │  RunPod    │  Modal    │  Replicate  │  Vast.ai     │    │
│  │  (Primary)   │  (Primary) │  (Alt)    │  (Alt)      │  (Budget)    │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Getting Started#

Prerequisites#

  1. Provider API key (RunComfy, RunPod, or other)
  2. Node.js 18+ environment
  3. Network access to provider APIs

Quick Start#

typescript
import { ComfyUIService } from '@oshun/comfyui-service';

// Initialize service
const service = new ComfyUIService({
  enableFailover: true,
  runcomfyConfig: {
    apiKey: process.env.RUNCOMFY_API_KEY,
    maxConcurrentJobs: 5,
  },
  runpodConfig: {
    apiKey: process.env.RUNPOD_API_KEY,
    endpointId: process.env.RUNPOD_COMFYUI_ENDPOINT,
  },
});

await service.initialize();

// Text-to-image generation
const result = await service.generateTxt2Img({
  prompt: 'A majestic dragon flying over a castle at sunset',
  negativePrompt: 'blurry, low quality',
  model: 'sd_xl_base_1.0.safetensors',
  width: 1024,
  height: 1024,
  steps: 30,
  cfg: 7.5,
});

console.log('Generated image:', result.outputs[0].url);

Custom Workflow#

typescript
import { RunComfyProvider } from '@oshun/comfy-provider';

const provider = new RunComfyProvider({
  apiKey: process.env.RUNCOMFY_API_KEY,
});

// Submit custom workflow
const result = await provider.submitJob({
  workflow: myCustomWorkflow,
  inputs: [
    { name: 'prompt', value: 'A beautiful landscape' },
    { name: 'seed', value: 12345 },
  ],
  webhookUrl: 'https://api.myapp.com/webhooks/comfyui',
});

Supported Generation Types#

Type Description Typical Latency
txt2img Text to image 20-60s
img2img Image transformation 15-45s
inpaint Fill masked areas 20-60s
upscale AI upscaling 10-30s
workflow Custom workflows Variable

Supported Models#

Stable Diffusion#

Model VRAM Resolution Notes
SD 1.5 4GB 512x512 Fast, lower quality
SD 2.1 5GB 768x768 Good quality
SDXL Base 8GB 1024x1024 High quality
SDXL + Refiner 12GB 1024x1024 Best quality

Flux#

Model VRAM Resolution Notes
Flux.1 Dev 24GB Variable Excellent quality
Flux.1 Schnell 20GB Variable Fast variant

Providers#

  • Pre-configured ComfyUI environments
  • Managed custom nodes
  • Pay-per-use pricing
  • Best for: Production workloads

RunPod#

  • GPU-based serverless
  • Full control over environment
  • Scale to zero capability
  • Best for: Custom setups

Multi-Provider Setup#

typescript
import { ComfyCloudProvider } from '@oshun/comfy-cloud';

const provider = new ComfyCloudProvider({
  providers: ['runcomfy', 'runpod', 'modal'],
  routingStrategy: 'cost_optimized',
  enableFailover: true,
  dailyBudget: 50,
});

Monitoring#

Key Metrics#

  • Job Success Rate: Target >99%
  • Average Latency: Target <30s for standard generation
  • Queue Depth: Monitor for scaling needs
  • Error Distribution: Track by error code

Health Checks#

typescript
const health = await service.healthCheck();

console.log('Providers:', health.providers);
console.log('Queue depth:', health.queue.depth);
console.log('Overall health:', health.healthy);

Best Practices#

  1. Use appropriate models: Match model to task complexity
  2. Optimize steps: 20-30 steps sufficient for most cases
  3. Batch similar jobs: Group by model to reduce loading
  4. Cache workflows: Template workflows for common tasks
  5. Monitor costs: Track GPU usage and optimize

Error Handling#

typescript
try {
  const result = await service.generateTxt2Img(params);
} catch (error) {
  if (error.code === 'RATE_LIMITED') {
    // Retry with exponential backoff
  } else if (error.code === 'TIMEOUT') {
    // Consider reducing complexity
  } else if (error.code === 'MODEL_NOT_FOUND') {
    // Check model availability
  }
}

Support#

  • Internal: #image-gen Slack channel
  • RunComfy: support@runcomfy.com
  • RunPod: RunPod Discord