Terraform configuration for the Iris AI Platform production environment on AWS EKS.
Overview#
This configuration provisions production-grade infrastructure:
- VPC: Multi-AZ network with public, private, and database subnets (3 NAT gateways)
- EKS: Kubernetes cluster with ON_DEMAND managed node groups
- RDS: PostgreSQL 16 with pgvector, Multi-AZ, read replica
- ElastiCache: Redis 7.1 replication group (3 clusters)
- S3: Encrypted buckets with cross-region replication
- ECR: Container registries with immutable tags
- IAM: IRSA roles for Kubernetes workloads
- Blue-Green: CodeDeploy for zero-downtime deployments
- Auto-Scaling: Cluster Autoscaler and HPA
- Services: API, conversation, memory, agent, voice, and worker runtimes
Prerequisites#
- AWS CLI configured with production credentials
- Terraform >= 1.10.0
- S3 bucket for Terraform state (
oshun-terraform-state) - DynamoDB table for state locking (
oshun-terraform-locks) - Production approval permissions
Production Configuration#
| Component | Production Configuration |
|---|---|
| EKS Nodes | ON_DEMAND instances |
| NAT Gateways | 3 (one per AZ for HA) |
| RDS | db.r6g.large, Multi-AZ + Replica |
| ElastiCache | cache.r6g.large, 3 clusters |
| Node Scaling | 3-15 nodes |
| HPA | 3-20 replicas |
| Deletion Protection | Enabled |
| Image Mutability | Immutable |
Deployment Process#
1. Infrastructure Changes (Requires Approval)#
bash
# Plan changes
terraform plan -out=tfplan
# Apply with approval
terraform apply tfplan
2. Application Deployment#
Production deployments use the GitHub Actions workflow with:
- Manual Trigger: Requires version input
- Approval Gate: Requires production environment approval
- Image Validation: Confirms the requested tag exists in all six ECR repositories
- Canary Analysis: Starts a shadow API canary and evaluates health and metrics
- Blue-Green Rollout: Applies the production Kubernetes overlay with immutable images
- Smoke Tests: Automated health verification
- Automatic Rollback: Reverts every Iris deployment on smoke-test failure
bash
# Trigger via GitHub CLI
gh workflow run iris-production-deploy.yml \
-f version=v1.2.3 \
-f deploy_infra=false
3. Emergency Rollback#
bash
# Rollback to previous version
gh workflow run iris-production-deploy.yml \
-f rollback_version=v1.2.2 \
-f skip_canary=true
Blue-Green Deployment#
The production environment uses CodeDeploy for blue-green deployments:
- Blue Environment: Current production
- Green Environment: New version
- Traffic Shift: Linear 10% per minute
- Validation: Lambda hook validates new environment
- Termination: Old environment terminated after 60 minutes
text
[Blue (v1.2.2)] ──┐
├──> [ALB] ──> Users
[Green (v1.2.3)]──┘
Traffic: 100% Blue → 90/10 → 80/20 → ... → 100% Green
Auto-Scaling#
Cluster Autoscaler#
- Scale nodes based on pending pods
- Scale-down delay: 10 minutes
- Expander: Least waste
Horizontal Pod Autoscaler#
- Services:
iris-api,iris-conversation,iris-memory,iris-agent,iris-voice,iris-worker - CPU threshold: 70%
- Memory threshold: 80%
- Min replicas: 3
- Max replicas: 20
- Scale-down stabilization: 5 minutes
Disaster Recovery#
| Metric | Target |
|---|---|
| RPO | 1 hour |
| RTO | 4 hours |
DR Components#
- RDS automated backups (30 days)
- RDS read replica for fast failover
- S3 cross-region replication to us-west-2
- ElastiCache snapshots (7 days)
- EKS cluster configuration in Git
Failover Procedure#
- Promote RDS read replica in DR region
- Update Route53 to DR region ALB
- Deploy from ECR replicated images
- Verify smoke tests pass
Security#
- All data encrypted at rest (KMS)
- All data encrypted in transit (TLS 1.2+)
- VPC endpoints for AWS services
- Network policies for pod isolation
- IRSA for fine-grained IAM
- GuardDuty for threat detection
- AWS Config for compliance
- Immutable container images
- Secrets in AWS Secrets Manager
Monitoring#
CloudWatch Dashboards#
- EKS cluster metrics
- RDS Performance Insights
- ElastiCache metrics
- Application metrics
Alarms#
| Alarm | Threshold | Action |
|---|---|---|
| RDS CPU | > 80% | SNS + Scale |
| RDS Connections | > 400 | SNS |
| ElastiCache Memory | > 80% | SNS |
| Smoke Test Failure | > 0 | SNS + Page |
| 5XX Error Rate | > 1% | Rollback |
| Deployment Failure | Any | Rollback |
Cost Optimization#
While production prioritizes reliability over cost:
- Reserved instances for predictable workloads
- Savings Plans for compute
- S3 Intelligent Tiering for long-term storage
- Right-sizing recommendations from Cost Explorer
Runbooks#
Scale Cluster Manually#
bash
# Update desired capacity
aws eks update-nodegroup-config \
--cluster-name iris-prod \
--nodegroup-name iris-prod-general \
--scaling-config minSize=5,maxSize=20,desiredSize=10
Force Rollback#
bash
# Get previous revision
kubectl rollout history deployment/iris-api -n iris
# Rollback to specific revision
kubectl rollout undo deployment/iris-api -n iris --to-revision=2
# Rollback all Iris service deployments to their previous revisions
for service in api conversation memory agent voice worker; do
kubectl rollout undo deployment/iris-${service} -n iris
done
Database Failover#
bash
# Promote read replica (creates new instance)
aws rds promote-read-replica \
--db-instance-identifier iris-prod-postgres-replica
# Update application configuration
kubectl set env deployment/iris-api \
DATABASE_HOST=new-primary-endpoint \
-n iris
Compliance#
- SOC 2 Type II logging enabled
- HIPAA-eligible services configured
- PCI DSS network isolation
- GDPR data residency controls
Support#
- On-Call: PagerDuty rotation
- Escalation: #iris-production Slack channel
- Runbooks: Confluence/iris-runbooks
Architecture#
text
┌─────────────────────────────────────────────────────────┐
│ Route53 │
└─────────────────────┬───────────────────────────────────┘
│
┌─────────────────────▼───────────────────────────────────┐
│ CloudFront │
└─────────────────────┬───────────────────────────────────┘
│
┌─────────────────────────────────────────┼─────────────────────────────────────┐
│ VPC (10.120.0.0/16) │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Public Subnets (3 AZs) │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌───────────────────────────┐ │ │
│ │ │ NAT GW │ │ NAT GW │ │ NAT GW │ │ Application Load │ │ │
│ │ │ (AZ-A) │ │ (AZ-B) │ │ (AZ-C) │ │ Balancer │ │ │
│ │ └─────────┘ └─────────┘ └─────────┘ └───────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Private Subnets (3 AZs) │ │
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │
│ │ │ EKS Node Group │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ iris-api │ │convers. │ │ memory │ │ agent │ │ │ │
│ │ │ │ (3-20) │ │ (3-20) │ │ (3-20) │ │ (3-20) │ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ voice │ │ worker │ │ │ │
│ │ │ │ (3-20) │ │ (3-20) │ │ │ │
│ │ │ └──────────┘ └──────────┘ │ │ │
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
│ ┌────────────────────────────────────────────────────────────────────────┐ │
│ │ Database Subnets (Isolated) │ │
│ │ ┌────────────────────────────┐ ┌────────────────────────────────┐ │ │
│ │ │ RDS PostgreSQL │ │ ElastiCache Redis │ │ │
│ │ │ Primary (Multi-AZ) │ │ Primary + 2 Replicas │ │ │
│ │ │ + │ │ │ │ │
│ │ │ Read Replica │ │ │ │ │
│ │ └────────────────────────────┘ └────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ DR Region (us-west-2) │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ S3 Replica │ │ ECR Replica │ │ RDS Snapshot│ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────┘