# Oshun ECS Architecture This document describes the Amazon ECS (Elastic Container Service) architecture used to run Oshun's containerized services. ## Overview Oshun uses AWS ECS with Fargate for running containerized microservices. The architecture prioritizes reliability, scalability, and cost efficiency. ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ AWS Cloud │ │ ┌───────────────────────────────────────────────────────────────────────┐ │ │ │ VPC (10.x.0.0/16) │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ │ │ Public Subnets │ │ │ │ │ │ ┌─────────────┐ ┌─────────────────────────────────────────┐ │ │ │ │ │ │ │ NAT Gateway │ │ Application Load Balancer │ │ │ │ │ │ │ └─────────────┘ │ (HTTPS → HTTP, WAF, SSL Termination) │ │ │ │ │ │ │ └──────────────────┬──────────────────────┘ │ │ │ │ │ └───────────────────────────────────────│─────────────────────────┘ │ │ │ │ │ │ │ │ │ ┌───────────────────────────────────────│─────────────────────────┐ │ │ │ │ │ Private Subnets │ │ │ │ │ │ │ ┌────────────────────────────────────┴───────────────────────┐ │ │ │ │ │ │ │ ECS Cluster │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ │ │ │ │ API │ │ Worker │ │ Frontend │ │ │ │ │ │ │ │ │ │ Service │ │ Service │ │ Service │ │ │ │ │ │ │ │ │ │ (Fargate) │ │ (Fargate) │ │ (Fargate) │ │ │ │ │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ └────────────────────────────────────────────────────────────┘ │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ └───────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## Components ### ECS Cluster The ECS cluster uses AWS Fargate as the compute engine, eliminating the need to manage EC2 instances. **Location:** `infra/terraform/ecs/` **Key features:** - Container Insights enabled for enhanced monitoring - CloudWatch log groups for container logs - Service discovery via AWS Cloud Map - Execute command enabled for debugging **Capacity providers:** | Environment | Providers | Usage | | ----------- | ------------- | ----------------------------- | | Staging | FARGATE, SPOT | Cost optimization with spot | | Production | FARGATE only | Reliability over cost savings | ### ECS Services Services run as Fargate tasks within the cluster. **Location:** `infra/terraform/ecs-services/` | Service | Port | CPU | Memory | Min Tasks | Max Tasks | | -------- | ---- | --- | ------ | --------- | --------- | | API | 3000 | 512 | 1024 | 2 | 10 | | Worker | 3001 | 512 | 1024 | 2 | 10 | | Frontend | 3002 | 256 | 512 | 2 | 6 | ### Networking **VPC Configuration:** - Staging: 2 Availability Zones, single NAT Gateway - Production: 3 Availability Zones, NAT Gateway per AZ **Subnets:** | Type | CIDR Pattern | Purpose | | ------- | --------------- | -------------------- | | Public | 10.x.1-3.0/24 | ALB, NAT Gateways | | Private | 10.x.11-13.0/24 | ECS tasks, databases | **Security Groups:** ``` ALB Security Group ├── Inbound: 443 from 0.0.0.0/0 (HTTPS) ├── Inbound: 80 from 0.0.0.0/0 (HTTP → redirect to HTTPS) └── Outbound: All traffic to VPC ECS Service Security Group ├── Inbound: Service port from ALB Security Group ├── Outbound: 443 to 0.0.0.0/0 (AWS APIs, external services) └── Outbound: Database ports to DB Security Group ``` ### Application Load Balancer **Location:** `infra/terraform/alb/` The ALB handles traffic routing, SSL termination, and health checks. **Features:** - HTTPS only (HTTP redirects to HTTPS) - ACM certificate for SSL/TLS - WAF integration for DDoS protection (production) - Access logs to S3 (production) - Cross-zone load balancing enabled **Target Groups:** Each service has a dedicated target group with: - Health check path: `/health` - Health check interval: 15-30 seconds - Deregistration delay: 30 seconds - Stickiness: Disabled (stateless services) ### Container Images Images are stored in Amazon ECR (Elastic Container Registry). **Location:** `infra/terraform/ecr/` **Repositories:** - `oshun-api` - API service - `oshun-worker` - Background worker service - `oshun-frontend` - Frontend service - `oshun-inference` - AI inference service **Image policies:** | Environment | Tag Mutability | Scan on Push | Max Images | | ----------- | -------------- | ------------ | ---------- | | Staging | MUTABLE | Yes | 10 | | Production | IMMUTABLE | Yes | 50 | ## Scaling ### Auto Scaling Services use target tracking scaling based on CPU utilization. **Configuration:** ```hcl # Staging autoscaling_min = 1 autoscaling_max = 3 autoscaling_cpu_target = 70 # Production autoscaling_min = 2 autoscaling_max = 10 autoscaling_cpu_target = 60 ``` **Scale-out behavior:** - Scale out: When average CPU > target for 3 consecutive minutes - Scale in: When average CPU < target - 10% for 15 minutes - Cooldown: 60 seconds between scaling actions ### Manual Scaling For expected traffic spikes: ```bash aws ecs update-service \ --cluster oshun-production \ --service oshun-api \ --desired-count 5 ``` ## Deployment ### Deployment Strategy **Rolling deployment (default):** ```hcl deployment_maximum_percent = 200 deployment_minimum_healthy_percent = 100 ``` This ensures zero-downtime deployments by: 1. Starting new tasks (up to 200% capacity) 2. Waiting for health checks to pass 3. Draining old tasks 4. Terminating old tasks **Blue/Green deployment (optional):** For services requiring instant rollback capability: ```hcl deployment_controller { type = "CODE_DEPLOY" } ``` See [deployment.md](./deployment.md) for details. ### Health Checks **ALB health checks:** ```hcl health_check { path = "/health" healthy_threshold = 2 unhealthy_threshold = 3 timeout = 5 interval = 15 matcher = "200" } ``` **ECS health checks:** ```dockerfile HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \ CMD curl -f http://localhost:3000/health || exit 1 ``` ## Monitoring ### CloudWatch Metrics **Container Insights metrics:** - `CpuUtilized` / `CpuReserved` - CPU usage - `MemoryUtilized` / `MemoryReserved` - Memory usage - `NetworkRxBytes` / `NetworkTxBytes` - Network I/O - `StorageReadBytes` / `StorageWriteBytes` - Disk I/O - `RunningTaskCount` - Number of running tasks ### CloudWatch Alarms **Location:** `infra/terraform/cloudwatch/` | Alarm | Threshold | Period | Action | | -------------------- | --------------- | ------ | --------------- | | High CPU | > 80% (staging) | 5 min | SNS alert | | High CPU | > 70% (prod) | 5 min | SNS + PagerDuty | | High Memory | > 80% | 5 min | SNS alert | | High Error Rate | > 1% (prod) | 5 min | SNS + PagerDuty | | High Latency | > 500ms (prod) | 5 min | SNS alert | | Unhealthy Host Count | > 0 | 1 min | SNS alert | ### Logging Container logs are sent to CloudWatch Logs. **Log groups:** - `/ecs/oshun-{environment}/{service}` - Service logs - `/ecs/oshun-{environment}/execution` - ECS agent logs **Log retention:** - Staging: 14 days - Production: 90 days **Querying logs:** ```bash # Recent API errors aws logs filter-log-events \ --log-group-name /ecs/oshun-production/api \ --filter-pattern "ERROR" \ --start-time $(date -d '1 hour ago' +%s000) # CloudWatch Logs Insights query aws logs start-query \ --log-group-name /ecs/oshun-production/api \ --start-time $(date -d '1 hour ago' +%s) \ --end-time $(date +%s) \ --query-string 'fields @timestamp, @message | filter @message like /ERROR/' ``` ## Security ### IAM Roles **Task execution role:** - Pull images from ECR - Write logs to CloudWatch - Retrieve secrets from Secrets Manager - Read parameters from SSM Parameter Store **Task role:** - Application-specific permissions - Access to S3 buckets - Access to SQS queues - X-Ray tracing (production) ### Secrets Management Secrets are stored in AWS Secrets Manager and injected as environment variables. ```hcl secrets = [ { name = "DATABASE_URL" valueFrom = "arn:aws:secretsmanager:us-east-1:123456789:secret:oshun/database-url" } ] ``` ### Network Security - Tasks run in private subnets (no public IPs) - Outbound internet via NAT Gateway - Security groups restrict traffic to required ports - VPC Flow Logs enabled for auditing ## Cost Optimization ### Staging Environment - Single NAT Gateway (vs. one per AZ) - FARGATE_SPOT capacity provider - Smaller task sizes (256 CPU, 512 Memory) - Shorter log retention (14 days) - No WAF or enhanced monitoring ### Production Environment - Reserved capacity (Savings Plans) for predictable baseline - Auto-scaling for variable load - Right-sized tasks based on actual usage - S3 Intelligent-Tiering for logs ### Cost Monitoring ```bash # Estimate monthly cost aws ce get-cost-forecast \ --time-period Start=$(date +%Y-%m-01),End=$(date -d 'next month' +%Y-%m-01) \ --granularity MONTHLY \ --metric UNBLENDED_COST \ --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Elastic Container Service"]}}' ``` ## Disaster Recovery ### Backup Strategy - Task definitions stored in Terraform (version controlled) - Container images in ECR with cross-region replication (optional) - Secrets replicated via Terraform (optional KMS cross-region) ### Recovery Procedures **Service failure:** 1. Auto-healing via ECS (unhealthy tasks replaced) 2. Auto-scaling for capacity issues 3. Manual scaling if needed **AZ failure:** 1. Tasks automatically redistributed across remaining AZs 2. ALB health checks stop routing to failed AZ 3. No manual intervention required (production) **Region failure:** 1. Deploy to secondary region using Terraform 2. Update DNS to point to new region 3. Restore data from cross-region backups ## Troubleshooting ### Common Issues **Task fails to start:** ```bash # Check stopped task reason aws ecs describe-tasks \ --cluster oshun-production \ --tasks # Common causes: # - Image pull failure (check ECR permissions) # - Resource constraints (increase CPU/memory) # - Health check failure (check /health endpoint) ``` **High CPU/Memory:** ```bash # Get service metrics aws cloudwatch get-metric-statistics \ --namespace AWS/ECS \ --metric-name CPUUtilization \ --dimensions Name=ClusterName,Value=oshun-production Name=ServiceName,Value=oshun-api \ --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \ --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \ --period 300 \ --statistics Average ``` **Deployment stuck:** ```bash # Check deployment status aws ecs describe-services \ --cluster oshun-production \ --services oshun-api \ --query 'services[0].deployments' # Force new deployment aws ecs update-service \ --cluster oshun-production \ --service oshun-api \ --force-new-deployment ``` ### Useful Commands ```bash # List all services aws ecs list-services --cluster oshun-production # Get service details aws ecs describe-services --cluster oshun-production --services oshun-api # List running tasks aws ecs list-tasks --cluster oshun-production --service-name oshun-api # Execute command in container aws ecs execute-command \ --cluster oshun-production \ --task \ --container api \ --interactive \ --command "/bin/sh" # Get task definition aws ecs describe-task-definition --task-definition oshun-api:latest ``` ## Related Documentation - [Deployment Process](./deployment.md) - [RunPod Integration](./runpod-integration.md) - [Incident Response](../reference/runbooks/incident-response.md) - [Scaling Runbook](../reference/runbooks/scaling.md)