# Oshun Deployment Process This document describes the deployment process for Oshun services, including CI/CD pipelines, rollback procedures, and best practices. ## Overview Oshun uses GitHub Actions for continuous integration and deployment: - **ECS Services:** Deployed via `.github/workflows/deploy-ecs.yml` - **RunPod Workers:** Deployed via `.github/workflows/deploy-runpod.yml` - **Infrastructure:** Managed via `.github/workflows/terraform.yml` ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Deployment Flow │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Commit │───▶│ Build │───▶│ Test │───▶│ Deploy │ │ │ │ to main │ │ & Lint │ │ & Scan │ │ Staging │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └──────┬──────┘ │ │ │ │ │ ▼ │ │ ┌─────────────┐ ┌─────────────┐ │ │ Deploy │◀───│ Approve │ │ │ Production │ │ (Manual) │ │ └─────────────┘ └─────────────┘ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## ECS Deployment ### Workflow Triggers The ECS deployment workflow triggers on: - **Push to main:** Automatic deployment to staging - **Release tags:** Deployment to production (with approval) - **Manual dispatch:** Select services and environment ### Deployment Stages #### 1. Planning ```yaml plan: - Detect changed services based on file paths - Determine deployment environment - Generate deployment matrix ``` Affected services are detected via path filters: | Path Pattern | Service | | ------------------------ | --------- | | `apps/*/api/**` | api | | `apps/*/worker/**` | worker | | `apps/*/frontend/**` | frontend | | `libs/*/ai-providers/**` | inference | #### 2. Build ```yaml build: - Login to AWS ECR - Build Docker image with BuildKit - Tag with commit SHA and 'latest' - Push to ECR - Scan for vulnerabilities ``` Build args: ```dockerfile --build-arg BUILD_ID=${{ github.run_id }} --build-arg COMMIT_SHA=${{ github.sha }} --build-arg BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ) ``` #### 3. Deploy Staging Staging deploys automatically without approval: ```yaml deploy-staging: - Register new task definition - Update ECS service - Wait for service stability - Run health checks ``` Deployment configuration: ```hcl deployment_maximum_percent = 200 deployment_minimum_healthy_percent = 100 ``` #### 4. Approve Production Production requires manual approval via GitHub Environments: 1. Reviewer receives notification 2. Reviews deployment summary 3. Approves or rejects deployment 4. Deployment proceeds or aborts Configure in: **Repository Settings > Environments > production-approval** - Add required reviewers - Optional: Wait timer (e.g., 10 minutes) - Optional: Deployment branches (main, release/\*) #### 5. Deploy Production After approval: ```yaml deploy-production: - Same steps as staging - More conservative deployment settings - Enhanced monitoring during rollout ``` Production deployment configuration: ```hcl deployment_maximum_percent = 150 # More conservative deployment_minimum_healthy_percent = 100 ``` ### Deployment Strategies #### Rolling Deployment (Default) ``` Time → ──────────────────────────────────────────────── v1 │████████████████████ │ v2 │ ████████████████████████████████ ──────────────────────────────────────────────── │ Start v2 │ v2 healthy │ Drain v1 │ Done ``` **Pros:** - Zero downtime - Gradual rollout - Automatic rollback on health check failure **Cons:** - Brief period with mixed versions - Slower rollout #### Blue/Green Deployment For services requiring instant rollback: ``` Time → ──────────────────────────────────────────────── Blue │████████████████████│ │ Green │ │████████████████████│ ──────────────────────────────────────────────── │ Deploy green │ Test │ Switch │ Drain │ ``` **Configuration:** ```hcl # In terraform deployment_controller { type = "CODE_DEPLOY" } ``` **Pros:** - Instant rollback (flip traffic back to blue) - Full testing before traffic switch - No mixed versions **Cons:** - Double capacity during deployment - More complex setup ### Health Checks Deployment waits for services to pass health checks: ```yaml health-check: - Wait for ECS service stability (5 min timeout) - Verify ALB target health - Run smoke tests against endpoints - Check CloudWatch for errors ``` Health check endpoints: | Service | Endpoint | Expected | | -------- | -------- | -------- | | API | /health | 200 OK | | Worker | /health | 200 OK | | Frontend | /health | 200 OK | ### Rollback Procedures #### Automatic Rollback ECS automatically rolls back if: - New tasks fail health checks - Tasks fail to start - Deployment timeout reached (10 min) #### Manual Rollback **Option 1: Redeploy previous version** ```bash # Find previous task definition aws ecs describe-services \ --cluster oshun-production \ --services oshun-api \ --query 'services[0].taskDefinition' # List recent task definitions aws ecs list-task-definitions \ --family-prefix oshun-api \ --sort DESC \ --max-items 5 # Redeploy previous version aws ecs update-service \ --cluster oshun-production \ --service oshun-api \ --task-definition oshun-api:42 # Previous version ``` **Option 2: Force new deployment (same version)** ```bash aws ecs update-service \ --cluster oshun-production \ --service oshun-api \ --force-new-deployment ``` **Option 3: CodeDeploy rollback (blue/green)** ```bash # Stop deployment aws deploy stop-deployment \ --deployment-id d-ABCDEF123 \ --auto-rollback-enabled # Manual rollback aws deploy create-deployment \ --application-name oshun-ecs-app \ --deployment-group-name oshun-api \ --revision '{"revisionType": "AppSpecContent", ...}' \ --description "Rollback to previous version" ``` ## RunPod Deployment ### Workflow Triggers - **Push to main:** Build and deploy changed images - **Manual dispatch:** Select specific images ### Deployment Stages #### 1. Change Detection ```yaml detect-changes: - base: docker/runpod/base/** - comfyui: docker/runpod/comfyui/** - sd: docker/runpod/sd/** - flux: docker/runpod/flux/** - inference: docker/runpod/inference/** ``` #### 2. Build Images ```yaml build: - Build base image first (if changed) - Build dependent images in parallel - Push to Docker Hub - Generate build summary ``` Image naming: ``` oshunai/runpod-{image-type}:{tag} Examples: - oshunai/runpod-base:latest - oshunai/runpod-comfyui-sdxl:v1.2.3 - oshunai/runpod-flux:abc1234 ``` #### 3. Update Endpoints After successful image push: ```yaml update-endpoints: - Authenticate with RunPod API - Update each endpoint's template - Verify endpoint status - Report results ``` ### Rollback Procedures **Option 1: Redeploy previous image** ```bash # Via GitHub Actions manual dispatch # Select image and specify previous tag ``` **Option 2: Update endpoint manually** 1. Go to RunPod dashboard 2. Select endpoint 3. Edit template 4. Change Docker image tag to previous version 5. Save changes ## Infrastructure Deployment ### Workflow Triggers - **Pull request:** Plan only (no apply) - **Push to main:** Apply to staging - **Manual approval:** Apply to production ### Deployment Stages #### 1. Change Detection ```yaml detect-changes: - environments/staging/** - environments/production/** - modules/** (affects both) ``` #### 2. Format & Validate ```yaml validate: - terraform fmt -check - terraform validate - tflint (optional) ``` #### 3. Plan ```yaml plan: - terraform init - terraform plan -out=tfplan - Post plan as PR comment - Upload plan artifact ``` #### 4. Apply ```yaml apply-staging: - Download plan artifact - terraform apply tfplan - Verify resources apply-production: - Requires manual approval - Same steps as staging ``` ### Rollback Procedures **Option 1: Revert and apply** ```bash # Revert problematic commit git revert git push origin main # CI/CD will apply the reverted state ``` **Option 2: Manual terraform apply** ```bash cd infra/terraform/environments/production terraform plan -target=module.affected_module terraform apply ``` **Option 3: State manipulation (emergency)** ```bash # Remove problematic resource from state terraform state rm aws_resource.name # Import correct resource terraform import aws_resource.name resource-id ``` ## Environment Variables ### Staging | Variable | Source | Example | | -------------- | --------------- | -------------------------- | | `NODE_ENV` | Task definition | staging | | `DATABASE_URL` | Secrets Manager | arn:aws:secretsmanager:... | | `REDIS_URL` | SSM Parameter | /oshun/staging/redis/url | | `LOG_LEVEL` | Task definition | debug | ### Production | Variable | Source | Example | | -------------- | --------------- | --------------------------- | | `NODE_ENV` | Task definition | production | | `DATABASE_URL` | Secrets Manager | arn:aws:secretsmanager:... | | `REDIS_URL` | SSM Parameter | /oshun/production/redis/url | | `LOG_LEVEL` | Task definition | info | ### Updating Secrets ```bash # Update secret value aws secretsmanager update-secret \ --secret-id oshun/production/database-url \ --secret-string "new-connection-string" # Force task restart to pick up new value aws ecs update-service \ --cluster oshun-production \ --service oshun-api \ --force-new-deployment ``` ## Monitoring Deployments ### GitHub Actions - Workflow runs visible in Actions tab - Deployment summary posted to Slack - Failed deployments trigger alerts ### AWS Console - ECS > Clusters > Services > Deployments tab - CloudWatch > Dashboards > Oshun-Production - CodeDeploy > Deployments (for blue/green) ### CLI Commands ```bash # Watch deployment progress watch -n 5 'aws ecs describe-services \ --cluster oshun-production \ --services oshun-api \ --query "services[0].deployments"' # Check running tasks aws ecs list-tasks \ --cluster oshun-production \ --service-name oshun-api # View task logs aws logs tail /ecs/oshun-production/api --follow ``` ## Best Practices ### Pre-Deployment Checklist - [ ] All tests passing in CI - [ ] Security scan clean - [ ] Changelog updated - [ ] Database migrations ready (if applicable) - [ ] Feature flags configured - [ ] Monitoring alerts reviewed ### During Deployment - [ ] Monitor CloudWatch dashboard - [ ] Watch error rates - [ ] Check service latency - [ ] Verify health check endpoints ### Post-Deployment Checklist - [ ] Verify all tasks healthy - [ ] Run smoke tests - [ ] Check key metrics - [ ] Notify stakeholders - [ ] Update deployment log ### Deployment Schedule | Time (UTC) | Action | | ------------- | ---------------------------------- | | Any time | Staging deployments | | Mon-Thu 14:00 | Production deployments (preferred) | | Fri | Avoid production deployments | | Weekends | Emergency only | ### Emergency Deployment For critical fixes outside normal hours: 1. Create hotfix branch from main 2. Apply minimal fix 3. Get approval from on-call engineer 4. Deploy using manual workflow dispatch 5. Monitor closely for 30 minutes 6. Document incident ## Troubleshooting ### Deployment Stuck ```bash # Check deployment events aws ecs describe-services \ --cluster oshun-production \ --services oshun-api \ --query 'services[0].events[:5]' # Common causes: # - Task failing health checks # - Resource constraints # - Image pull failures ``` ### Tasks Not Starting ```bash # Check stopped tasks aws ecs list-tasks \ --cluster oshun-production \ --service-name oshun-api \ --desired-status STOPPED # Get stop reason aws ecs describe-tasks \ --cluster oshun-production \ --tasks \ --query 'tasks[0].stoppedReason' ``` ### Health Check Failures ```bash # Test health endpoint curl -v https://api.oshun.ai/health # Check ALB target health aws elbv2 describe-target-health \ --target-group-arn ``` ## Related Documentation - [ECS Architecture](./ecs-architecture.md) - [RunPod Integration](./runpod-integration.md) - [Incident Response Runbook](../reference/runbooks/incident-response.md) - [Scaling Runbook](../reference/runbooks/scaling.md)