Disciplines · Reference

Scaling Runbook

Oshun uses multiple scaling mechanisms:

8sections3 minread
Markdown source
On this page

This runbook provides guidance for scaling Oshun services to handle increased load or optimize costs during low-traffic periods.

Overview#

Oshun uses multiple scaling mechanisms:

Component Scaling Method Metric Default Config
ECS Services Auto Scaling CPU Utilization 60-80% target
RunPod Workers Auto Scaling Queue Depth 0 min, varies max
Database Manual (instance) CPU/Connections As needed
Cache (Redis) Manual (node type) Memory/Connections As needed

ECS Service Scaling#

Current Configuration#

bash
# View current scaling configuration
aws application-autoscaling describe-scalable-targets \
  --service-namespace ecs \
  --resource-ids service/oshun-production/oshun-api

# View scaling policies
aws application-autoscaling describe-scaling-policies \
  --service-namespace ecs \
  --resource-id service/oshun-production/oshun-api

Auto Scaling Settings#

Service Min Max Target CPU Scale Out Cooldown Scale In Cooldown
API 2 10 60% 60s 300s
Worker 2 10 60% 60s 300s
Frontend 2 6 70% 60s 300s

Manual Scaling#

Scale up immediately:

bash
# Increase desired count
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --desired-count 5

# Verify scaling
watch -n 5 'aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query "services[0].{Running:runningCount,Desired:desiredCount,Pending:pendingCount}"'

Adjust auto scaling limits:

bash
# Increase max capacity
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 2 \
  --max-capacity 20

Scale down (cost optimization):

bash
# During known low-traffic periods
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --desired-count 2

# Adjust min capacity for sustained low traffic
aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 1 \
  --max-capacity 10

Scaling Based on Custom Metrics#

Scale based on SQS queue depth:

bash
# For worker services processing queues
aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --resource-id service/oshun-production/oshun-worker \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-name sqs-queue-scaling \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 100,
    "CustomizedMetricSpecification": {
      "MetricName": "ApproximateNumberOfMessagesVisible",
      "Namespace": "AWS/SQS",
      "Dimensions": [{"Name": "QueueName", "Value": "oshun-production-generation-requests"}],
      "Statistic": "Average"
    },
    "ScaleOutCooldown": 60,
    "ScaleInCooldown": 300
  }'

Scheduled Scaling#

For predictable traffic patterns:

bash
# Scale up for business hours (9 AM UTC)
aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --scheduled-action-name scale-up-morning \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --schedule "cron(0 9 ? * MON-FRI *)" \
  --scalable-target-action MinCapacity=4,MaxCapacity=15

# Scale down for night (10 PM UTC)
aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --scheduled-action-name scale-down-night \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --schedule "cron(0 22 ? * * *)" \
  --scalable-target-action MinCapacity=2,MaxCapacity=6

RunPod Scaling#

Endpoint Configuration#

RunPod endpoints scale automatically based on queue depth.

View current configuration:

bash
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { endpoints { id name workersMax workersMin gpuIds } } }"}'

Scaling Parameters#

Endpoint Min Workers Max Workers Idle Timeout GPU Type
ComfyUI SD1.5 0 5 60s A40
ComfyUI SDXL 0 3 120s A100
Flux 0 3 120s A100
SD 0 10 30s 4090

Adjusting Max Workers#

Via RunPod Dashboard:

  1. Go to RunPod Console
  2. Select endpoint
  3. Click "Edit"
  4. Adjust "Max Workers"
  5. Save

Via GraphQL API:

bash
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{
    "query": "mutation { updateEndpoint(input: { id: \"ENDPOINT_ID\", workersMax: 10 }) { id workersMax } }"
  }'

Scaling for Expected Load#

Before a known high-traffic event:

  1. Increase max workers on all endpoints
  2. Consider keeping minimum workers warm (min > 0)
  3. Monitor queue depth during event
bash
# Set minimum workers to avoid cold starts
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{
    "query": "mutation { updateEndpoint(input: { id: \"ENDPOINT_ID\", workersMin: 2, workersMax: 10 }) { id workersMin workersMax } }"
  }'

After event:

  1. Reset minimum workers to 0
  2. Reduce max workers if not needed
  3. Review costs

Database Scaling#

RDS Scaling Options#

Scaling Type When to Use Downtime
Vertical (resize) CPU/memory consistently high Yes (minutes)
Storage autoscale Storage approaching capacity No
Read replicas Read-heavy workload No
Aurora Serverless Variable/unpredictable workload Migration req.

Vertical Scaling#

Check current utilization:

bash
# CPU utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name CPUUtilization \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 \
  --statistics Average Maximum

# Memory
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name FreeableMemory \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 \
  --statistics Average Minimum

Resize instance:

bash
# Schedule resize during maintenance window
aws rds modify-db-instance \
  --db-instance-identifier oshun-production \
  --db-instance-class db.r6g.xlarge \
  --apply-immediately false

# Or apply immediately (causes downtime)
aws rds modify-db-instance \
  --db-instance-identifier oshun-production \
  --db-instance-class db.r6g.xlarge \
  --apply-immediately true

Adding Read Replicas#

For read-heavy workloads:

bash
# Create read replica
aws rds create-db-instance-read-replica \
  --db-instance-identifier oshun-production-read1 \
  --source-db-instance-identifier oshun-production \
  --db-instance-class db.r6g.large

# Update application to use read replica for reads
# Configure in connection string or ORM settings

Connection Pooling#

If hitting connection limits:

  1. Enable connection pooling (PgBouncer)
  2. Adjust application pool settings
  3. Increase max_connections (requires restart)
bash
# Check current connections
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '1 day ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Maximum Average

Redis/ElastiCache Scaling#

Scaling Options#

Scaling Type When to Use Downtime
Vertical (resize) Memory consistently high Brief
Horizontal (shards) High throughput needed Depends
Read replicas Read-heavy cache workload No

Check Current Utilization#

bash
# Memory utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/ElastiCache \
  --metric-name DatabaseMemoryUsagePercentage \
  --dimensions Name=CacheClusterId,Value=oshun-production \
  --start-time $(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 \
  --statistics Average Maximum

# CPU utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/ElastiCache \
  --metric-name CPUUtilization \
  --dimensions Name=CacheClusterId,Value=oshun-production \
  --start-time $(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 3600 \
  --statistics Average Maximum

Resize Cache Node#

bash
# Modify node type
aws elasticache modify-cache-cluster \
  --cache-cluster-id oshun-production \
  --cache-node-type cache.r6g.large \
  --apply-immediately

Scaling Checklist#

Before Scaling Up#

  • Identify bottleneck (CPU, memory, connections, etc.)
  • Check if issue is load-related or code bug
  • Review recent deployments for potential issues
  • Estimate cost impact of scaling
  • Notify team in #ops channel

After Scaling#

  • Verify metrics improved
  • Monitor for 30 minutes
  • Document scaling action
  • Schedule review for potential permanent change

Cost Considerations#

Action Cost Impact Reversibility
ECS scale up (auto) Hourly cost increase Auto
ECS scale up (manual) Hourly cost increase Manual
RunPod max workers increase Per-second GPU cost Manual
RDS instance resize Hourly cost change Manual
RDS read replica Hourly cost add Manual
ElastiCache resize Hourly cost change Manual

Emergency Scaling Procedures#

Traffic Spike#

  1. Immediately increase ECS desired count:

    bash
    aws ecs update-service --cluster oshun-production --service oshun-api --desired-count 8

  2. Increase auto scaling max:

    bash
    aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --max-capacity 20

  3. If RunPod queues backing up, increase max workers

  4. Monitor CloudWatch dashboard closely

  5. Communicate status in #incidents

Resource Exhaustion#

  1. Scale horizontally (more instances) rather than vertically (larger instances)
  2. Enable request throttling if needed
  3. Consider circuit breaker patterns
  4. Shed non-critical load (e.g., analytics)