# Iris AI Platform - Production Infrastructure

Terraform configuration for the Iris AI Platform production environment on AWS
EKS.

## Overview

This configuration provisions production-grade infrastructure:

- **VPC**: Multi-AZ network with public, private, and database subnets (3 NAT
  gateways)
- **EKS**: Kubernetes cluster with ON_DEMAND managed node groups
- **RDS**: PostgreSQL 16 with pgvector, Multi-AZ, read replica
- **ElastiCache**: Redis 7.1 replication group (3 clusters)
- **S3**: Encrypted buckets with cross-region replication
- **ECR**: Container registries with immutable tags
- **IAM**: IRSA roles for Kubernetes workloads
- **Blue-Green**: CodeDeploy for zero-downtime deployments
- **Auto-Scaling**: Cluster Autoscaler and HPA
- **Services**: API, conversation, memory, agent, voice, and worker runtimes

## Prerequisites

1. AWS CLI configured with production credentials
2. Terraform >= 1.10.0
3. S3 bucket for Terraform state (`oshun-terraform-state`)
4. DynamoDB table for state locking (`oshun-terraform-locks`)
5. **Production approval permissions**

## Production Configuration

| Component           | Production Configuration         |
| ------------------- | -------------------------------- |
| EKS Nodes           | ON_DEMAND instances              |
| NAT Gateways        | 3 (one per AZ for HA)            |
| RDS                 | db.r6g.large, Multi-AZ + Replica |
| ElastiCache         | cache.r6g.large, 3 clusters      |
| Node Scaling        | 3-15 nodes                       |
| HPA                 | 3-20 replicas                    |
| Deletion Protection | Enabled                          |
| Image Mutability    | Immutable                        |

## Deployment Process

### 1. Infrastructure Changes (Requires Approval)

```bash
# Plan changes
terraform plan -out=tfplan

# Apply with approval
terraform apply tfplan
```

### 2. Application Deployment

Production deployments use the GitHub Actions workflow with:

1. **Manual Trigger**: Requires version input
2. **Approval Gate**: Requires production environment approval
3. **Image Validation**: Confirms the requested tag exists in all six ECR
   repositories
4. **Canary Analysis**: Starts a shadow API canary and evaluates health and
   metrics
5. **Blue-Green Rollout**: Applies the production Kubernetes overlay with
   immutable images
6. **Smoke Tests**: Automated health verification
7. **Automatic Rollback**: Reverts every Iris deployment on smoke-test failure

```bash
# Trigger via GitHub CLI
gh workflow run iris-production-deploy.yml \
  -f version=v1.2.3 \
  -f deploy_infra=false
```

### 3. Emergency Rollback

```bash
# Rollback to previous version
gh workflow run iris-production-deploy.yml \
  -f rollback_version=v1.2.2 \
  -f skip_canary=true
```

## Blue-Green Deployment

The production environment uses CodeDeploy for blue-green deployments:

1. **Blue Environment**: Current production
2. **Green Environment**: New version
3. **Traffic Shift**: Linear 10% per minute
4. **Validation**: Lambda hook validates new environment
5. **Termination**: Old environment terminated after 60 minutes

```
[Blue (v1.2.2)] ──┐
                  ├──> [ALB] ──> Users
[Green (v1.2.3)]──┘

Traffic: 100% Blue → 90/10 → 80/20 → ... → 100% Green
```

## Auto-Scaling

### Cluster Autoscaler

- Scale nodes based on pending pods
- Scale-down delay: 10 minutes
- Expander: Least waste

### Horizontal Pod Autoscaler

- Services: `iris-api`, `iris-conversation`, `iris-memory`, `iris-agent`,
  `iris-voice`, `iris-worker`
- CPU threshold: 70%
- Memory threshold: 80%
- Min replicas: 3
- Max replicas: 20
- Scale-down stabilization: 5 minutes

## Disaster Recovery

| Metric | Target  |
| ------ | ------- |
| RPO    | 1 hour  |
| RTO    | 4 hours |

### DR Components

- RDS automated backups (30 days)
- RDS read replica for fast failover
- S3 cross-region replication to us-west-2
- ElastiCache snapshots (7 days)
- EKS cluster configuration in Git

### Failover Procedure

1. Promote RDS read replica in DR region
2. Update Route53 to DR region ALB
3. Deploy from ECR replicated images
4. Verify smoke tests pass

## Security

- All data encrypted at rest (KMS)
- All data encrypted in transit (TLS 1.2+)
- VPC endpoints for AWS services
- Network policies for pod isolation
- IRSA for fine-grained IAM
- GuardDuty for threat detection
- AWS Config for compliance
- Immutable container images
- Secrets in AWS Secrets Manager

## Monitoring

### CloudWatch Dashboards

- EKS cluster metrics
- RDS Performance Insights
- ElastiCache metrics
- Application metrics

### Alarms

| Alarm              | Threshold | Action      |
| ------------------ | --------- | ----------- |
| RDS CPU            | > 80%     | SNS + Scale |
| RDS Connections    | > 400     | SNS         |
| ElastiCache Memory | > 80%     | SNS         |
| Smoke Test Failure | > 0       | SNS + Page  |
| 5XX Error Rate     | > 1%      | Rollback    |
| Deployment Failure | Any       | Rollback    |

## Cost Optimization

While production prioritizes reliability over cost:

- Reserved instances for predictable workloads
- Savings Plans for compute
- S3 Intelligent Tiering for long-term storage
- Right-sizing recommendations from Cost Explorer

## Runbooks

### Scale Cluster Manually

```bash
# Update desired capacity
aws eks update-nodegroup-config \
  --cluster-name iris-prod \
  --nodegroup-name iris-prod-general \
  --scaling-config minSize=5,maxSize=20,desiredSize=10
```

### Force Rollback

```bash
# Get previous revision
kubectl rollout history deployment/iris-api -n iris

# Rollback to specific revision
kubectl rollout undo deployment/iris-api -n iris --to-revision=2

# Rollback all Iris service deployments to their previous revisions
for service in api conversation memory agent voice worker; do
  kubectl rollout undo deployment/iris-${service} -n iris
done
```

### Database Failover

```bash
# Promote read replica (creates new instance)
aws rds promote-read-replica \
  --db-instance-identifier iris-prod-postgres-replica

# Update application configuration
kubectl set env deployment/iris-api \
  DATABASE_HOST=new-primary-endpoint \
  -n iris
```

## Compliance

- SOC 2 Type II logging enabled
- HIPAA-eligible services configured
- PCI DSS network isolation
- GDPR data residency controls

## Support

- **On-Call**: PagerDuty rotation
- **Escalation**: #iris-production Slack channel
- **Runbooks**: Confluence/iris-runbooks

## Architecture

```
                    ┌─────────────────────────────────────────────────────────┐
                    │                     Route53                              │
                    └─────────────────────┬───────────────────────────────────┘
                                          │
                    ┌─────────────────────▼───────────────────────────────────┐
                    │                   CloudFront                             │
                    └─────────────────────┬───────────────────────────────────┘
                                          │
┌─────────────────────────────────────────┼─────────────────────────────────────┐
│                         VPC (10.120.0.0/16)                                   │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                    Public Subnets (3 AZs)                              │  │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌───────────────────────────┐  │  │
│  │  │ NAT GW  │  │ NAT GW  │  │ NAT GW  │  │   Application Load        │  │  │
│  │  │  (AZ-A) │  │  (AZ-B) │  │  (AZ-C) │  │   Balancer                │  │  │
│  │  └─────────┘  └─────────┘  └─────────┘  └───────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                    Private Subnets (3 AZs)                             │  │
│  │  ┌──────────────────────────────────────────────────────────────────┐  │  │
│  │  │                      EKS Node Group                              │  │  │
│  │  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐            │  │  │
│  │  │  │ iris-api │ │convers.  │ │ memory   │ │ agent    │            │  │  │
│  │  │  │ (3-20)   │ │ (3-20)   │ │ (3-20)   │ │ (3-20)   │            │  │  │
│  │  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘            │  │  │
│  │  │  ┌──────────┐ ┌──────────┐                                      │  │  │
│  │  │  │ voice    │ │ worker   │                                      │  │  │
│  │  │  │ (3-20)   │ │ (3-20)   │                                      │  │  │
│  │  │  └──────────┘ └──────────┘                                      │  │  │
│  │  └──────────────────────────────────────────────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                   Database Subnets (Isolated)                          │  │
│  │  ┌────────────────────────────┐  ┌────────────────────────────────┐   │  │
│  │  │    RDS PostgreSQL          │  │   ElastiCache Redis            │   │  │
│  │  │    Primary (Multi-AZ)      │  │   Primary + 2 Replicas         │   │  │
│  │  │         +                  │  │                                │   │  │
│  │  │    Read Replica            │  │                                │   │  │
│  │  └────────────────────────────┘  └────────────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────────────────────┘

                    ┌─────────────────────────────────────────────────────────┐
                    │                DR Region (us-west-2)                     │
                    │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐      │
                    │  │ S3 Replica  │  │ ECR Replica │  │ RDS Snapshot│      │
                    │  └─────────────┘  └─────────────┘  └─────────────┘      │
                    └─────────────────────────────────────────────────────────┘
```
