Oshun Deployment Process — Infrastructure

This document describes the deployment process for Oshun services, including CI/CD pipelines, rollback procedures, and best practices.

Overview#

Oshun uses GitHub Actions for continuous integration and deployment:

ECS Services: Deployed via .github/workflows/deploy-ecs.yml
RunPod Workers: Deployed via .github/workflows/deploy-runpod.yml
Infrastructure: Managed via .github/workflows/terraform.yml

text

┌─────────────────────────────────────────────────────────────────────────────┐
│                          Deployment Flow                                     │
│                                                                              │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐  │
│  │   Commit    │───▶│    Build    │───▶│    Test     │───▶│   Deploy    │  │
│  │   to main   │    │   & Lint    │    │   & Scan    │    │   Staging   │  │
│  └─────────────┘    └─────────────┘    └─────────────┘    └──────┬──────┘  │
│                                                                   │         │
│                                                                   ▼         │
│                                            ┌─────────────┐    ┌─────────────┐
│                                            │   Deploy    │◀───│   Approve   │
│                                            │  Production │    │   (Manual)  │
│                                            └─────────────┘    └─────────────┘
└─────────────────────────────────────────────────────────────────────────────┘

ECS Deployment#

Workflow Triggers#

The ECS deployment workflow triggers on:

Push to main: Automatic deployment to staging
Release tags: Deployment to production (with approval)
Manual dispatch: Select services and environment

Deployment Stages#

1. Planning#

yaml

plan:
  - Detect changed services based on file paths
  - Determine deployment environment
  - Generate deployment matrix

Affected services are detected via path filters:

Path Pattern	Service
`apps//api/*`	api
`apps//worker/*`	worker
`apps//frontend/*`	frontend
`libs//ai-providers/*`	inference

2. Build#

yaml

build:
  - Login to AWS ECR
  - Build Docker image with BuildKit
  - Tag with commit SHA and 'latest'
  - Push to ECR
  - Scan for vulnerabilities

Build args:

dockerfile

--build-arg BUILD_ID=${{ github.run_id }}
--build-arg COMMIT_SHA=${{ github.sha }}
--build-arg BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)

3. Deploy Staging#

Staging deploys automatically without approval:

yaml

deploy-staging:
  - Register new task definition
  - Update ECS service
  - Wait for service stability
  - Run health checks

Deployment configuration:

hcl

deployment_maximum_percent         = 200
deployment_minimum_healthy_percent = 100

4. Approve Production#

Production requires manual approval via GitHub Environments:

Reviewer receives notification
Reviews deployment summary
Approves or rejects deployment
Deployment proceeds or aborts

Configure in: Repository Settings > Environments > production-approval

Add required reviewers
Optional: Wait timer (e.g., 10 minutes)
Optional: Deployment branches (main, release/*)

5. Deploy Production#

After approval:

yaml

deploy-production:
  - Same steps as staging
  - More conservative deployment settings
  - Enhanced monitoring during rollout

Production deployment configuration:

hcl

deployment_maximum_percent         = 150  # More conservative
deployment_minimum_healthy_percent = 100

Deployment Strategies#

Rolling Deployment (Default)#

text

Time →
────────────────────────────────────────────────
v1 │████████████████████                      │
v2 │            ████████████████████████████████
────────────────────────────────────────────────
     │ Start v2  │ v2 healthy │ Drain v1  │ Done

Pros:

Zero downtime
Gradual rollout
Automatic rollback on health check failure

Cons:

Brief period with mixed versions
Slower rollout

Blue/Green Deployment#

For services requiring instant rollback:

text

Time →
────────────────────────────────────────────────
Blue  │████████████████████│                    │
Green │                    │████████████████████│
────────────────────────────────────────────────
       │ Deploy green │ Test │ Switch │ Drain │

Configuration:

hcl

# In terraform
deployment_controller {
  type = "CODE_DEPLOY"
}

Pros:

Instant rollback (flip traffic back to blue)
Full testing before traffic switch
No mixed versions

Cons:

Double capacity during deployment
More complex setup

Health Checks#

Deployment waits for services to pass health checks:

yaml

health-check:
  - Wait for ECS service stability (5 min timeout)
  - Verify ALB target health
  - Run smoke tests against endpoints
  - Check CloudWatch for errors

Health check endpoints:

Service	Endpoint	Expected
API	/health	200 OK
Worker	/health	200 OK
Frontend	/health	200 OK

Rollback Procedures#

Automatic Rollback#

ECS automatically rolls back if:

New tasks fail health checks
Tasks fail to start
Deployment timeout reached (10 min)

Manual Rollback#

Option 1: Redeploy previous version

bash

# Find previous task definition
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query 'services[0].taskDefinition'

# List recent task definitions
aws ecs list-task-definitions \
  --family-prefix oshun-api \
  --sort DESC \
  --max-items 5

# Redeploy previous version
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --task-definition oshun-api:42  # Previous version

Option 2: Force new deployment (same version)

bash

aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --force-new-deployment

Option 3: CodeDeploy rollback (blue/green)

bash

# Stop deployment
aws deploy stop-deployment \
  --deployment-id d-ABCDEF123 \
  --auto-rollback-enabled

# Manual rollback
aws deploy create-deployment \
  --application-name oshun-ecs-app \
  --deployment-group-name oshun-api \
  --revision '{"revisionType": "AppSpecContent", ...}' \
  --description "Rollback to previous version"

RunPod Deployment#

Workflow Triggers#

Push to main: Build and deploy changed images
Manual dispatch: Select specific images

Deployment Stages#

1. Change Detection#

yaml

detect-changes:
  - base: docker/runpod/base/**
  - comfyui: docker/runpod/comfyui/**
  - sd: docker/runpod/sd/**
  - flux: docker/runpod/flux/**
  - inference: docker/runpod/inference/**

2. Build Images#

yaml

build:
  - Build base image first (if changed)
  - Build dependent images in parallel
  - Push to Docker Hub
  - Generate build summary

Image naming:

text

oshunai/runpod-{image-type}:{tag}

Examples:
- oshunai/runpod-base:latest
- oshunai/runpod-comfyui-sdxl:v1.2.3
- oshunai/runpod-flux:abc1234

3. Update Endpoints#

After successful image push:

yaml

update-endpoints:
  - Authenticate with RunPod API
  - Update each endpoint's template
  - Verify endpoint status
  - Report results

Rollback Procedures#

Option 1: Redeploy previous image

bash

# Via GitHub Actions manual dispatch
# Select image and specify previous tag

Option 2: Update endpoint manually

Go to RunPod dashboard
Select endpoint
Edit template
Change Docker image tag to previous version
Save changes

Infrastructure Deployment#

Workflow Triggers#

Pull request: Plan only (no apply)
Push to main: Apply to staging
Manual approval: Apply to production

Deployment Stages#

1. Change Detection#

yaml

detect-changes:
  - environments/staging/**
  - environments/production/**
  - modules/** (affects both)

2. Format & Validate#

yaml

validate:
  - terraform fmt -check
  - terraform validate
  - tflint (optional)

3. Plan#

yaml

plan:
  - terraform init
  - terraform plan -out=tfplan
  - Post plan as PR comment
  - Upload plan artifact

4. Apply#

yaml

apply-staging:
  - Download plan artifact
  - terraform apply tfplan
  - Verify resources

apply-production:
  - Requires manual approval
  - Same steps as staging

Rollback Procedures#

Option 1: Revert and apply

bash

# Revert problematic commit
git revert <commit-sha>
git push origin main

# CI/CD will apply the reverted state

Option 2: Manual terraform apply

bash

cd infra/terraform/environments/production
terraform plan -target=module.affected_module
terraform apply

Option 3: State manipulation (emergency)

bash

# Remove problematic resource from state
terraform state rm aws_resource.name

# Import correct resource
terraform import aws_resource.name resource-id

Environment Variables#

Staging#

Variable	Source	Example
`NODE_ENV`	Task definition	staging
`DATABASE_URL`	Secrets Manager	arn:aws:secretsmanager:...
`REDIS_URL`	SSM Parameter	/oshun/staging/redis/url
`LOG_LEVEL`	Task definition	debug

Production#

Variable	Source	Example
`NODE_ENV`	Task definition	production
`DATABASE_URL`	Secrets Manager	arn:aws:secretsmanager:...
`REDIS_URL`	SSM Parameter	/oshun/production/redis/url
`LOG_LEVEL`	Task definition	info

Updating Secrets#

bash

# Update secret value
aws secretsmanager update-secret \
  --secret-id oshun/production/database-url \
  --secret-string "new-connection-string"

# Force task restart to pick up new value
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --force-new-deployment

Monitoring Deployments#

GitHub Actions#

Workflow runs visible in Actions tab
Deployment summary posted to Slack
Failed deployments trigger alerts

AWS Console#

ECS > Clusters > Services > Deployments tab
CloudWatch > Dashboards > Oshun-Production
CodeDeploy > Deployments (for blue/green)

CLI Commands#

bash

# Watch deployment progress
watch -n 5 'aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query "services[0].deployments"'

# Check running tasks
aws ecs list-tasks \
  --cluster oshun-production \
  --service-name oshun-api

# View task logs
aws logs tail /ecs/oshun-production/api --follow

Best Practices#

Pre-Deployment Checklist#

All tests passing in CI
Security scan clean
Changelog updated
Database migrations ready (if applicable)
Feature flags configured
Monitoring alerts reviewed

During Deployment#

Monitor CloudWatch dashboard
Watch error rates
Check service latency
Verify health check endpoints

Post-Deployment Checklist#

Verify all tasks healthy
Run smoke tests
Check key metrics
Notify stakeholders
Update deployment log

Deployment Schedule#

Time (UTC)	Action
Any time	Staging deployments
Mon-Thu 14:00	Production deployments (preferred)
Fri	Avoid production deployments
Weekends	Emergency only

Emergency Deployment#

For critical fixes outside normal hours:

Create hotfix branch from main
Apply minimal fix
Get approval from on-call engineer
Deploy using manual workflow dispatch
Monitor closely for 30 minutes
Document incident

Troubleshooting#

Deployment Stuck#

bash

# Check deployment events
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query 'services[0].events[:5]'

# Common causes:
# - Task failing health checks
# - Resource constraints
# - Image pull failures

Tasks Not Starting#

bash

# Check stopped tasks
aws ecs list-tasks \
  --cluster oshun-production \
  --service-name oshun-api \
  --desired-status STOPPED

# Get stop reason
aws ecs describe-tasks \
  --cluster oshun-production \
  --tasks <task-id> \
  --query 'tasks[0].stoppedReason'

Health Check Failures#

bash

# Test health endpoint
curl -v https://api.oshun.ai/health

# Check ALB target health
aws elbv2 describe-target-health \
  --target-group-arn <target-group-arn>

Overview#

ECS Deployment#

Workflow Triggers#

Deployment Stages#

1. Planning#

2. Build#

3. Deploy Staging#

4. Approve Production#

5. Deploy Production#

Deployment Strategies#

Rolling Deployment (Default)#

Blue/Green Deployment#

Health Checks#

Rollback Procedures#

Automatic Rollback#

Manual Rollback#

RunPod Deployment#

Workflow Triggers#

Deployment Stages#

1. Change Detection#

2. Build Images#

3. Update Endpoints#

Rollback Procedures#

Infrastructure Deployment#

Workflow Triggers#

Deployment Stages#

1. Change Detection#

2. Format & Validate#

3. Plan#

4. Apply#

Rollback Procedures#

Environment Variables#

Staging#

Production#

Updating Secrets#

Monitoring Deployments#

GitHub Actions#

AWS Console#

CLI Commands#

Best Practices#

Pre-Deployment Checklist#

During Deployment#

Post-Deployment Checklist#

Deployment Schedule#

Emergency Deployment#

Troubleshooting#

Deployment Stuck#

Tasks Not Starting#

Health Check Failures#

Related Documentation#