# Psyche AI Platform - Terraform Infrastructure

This Terraform module deploys the complete infrastructure for the Psyche AI
Virtual Assistant Platform on AWS, including GPU-accelerated EKS clusters,
PostgreSQL with pgvector, Redis caching, S3 storage, and CloudFront CDN.

## Architecture Overview

```
                                    ┌─────────────────────────────────────────────────────────────────┐
                                    │                          AWS Cloud                               │
                                    │  ┌─────────────────────────────────────────────────────────────┐│
                                    │  │                           VPC                                ││
                                    │  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         ││
                                    │  │  │  Public     │  │  Public     │  │  Public     │         ││
┌──────────┐  ┌──────────────┐     │  │  │  Subnet     │  │  Subnet     │  │  Subnet     │         ││
│  Users   │─▶│  CloudFront  │─────┼──┼──│  (ALB/NLB)  │  │  (ALB/NLB)  │  │  (ALB/NLB)  │         ││
└──────────┘  └──────────────┘     │  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘         ││
                    │              │  │         │                │                │                ││
                    │              │  │  ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐         ││
                    │              │  │  │  Private    │  │  Private    │  │  Private    │         ││
                    │              │  │  │  Subnet     │  │  Subnet     │  │  Subnet     │         ││
                    │              │  │  │             │  │             │  │             │         ││
                    │              │  │  │ ┌─────────┐ │  │ ┌─────────┐ │  │ ┌─────────┐ │         ││
                    │              │  │  │ │EKS Node │ │  │ │EKS Node │ │  │ │EKS Node │ │         ││
                    │              │  │  │ │(General)│ │  │ │(General)│ │  │ │(General)│ │         ││
                    │              │  │  │ └─────────┘ │  │ └─────────┘ │  │ └─────────┘ │         ││
                    │              │  │  │ ┌─────────┐ │  │ ┌─────────┐ │  │             │         ││
                    │              │  │  │ │GPU Node │ │  │ │GPU Node │ │  │  Karpenter  │         ││
                    │              │  │  │ │(NVIDIA) │ │  │ │(NVIDIA) │ │  │  Auto-scale │         ││
                    │              │  │  │ └─────────┘ │  │ └─────────┘ │  │             │         ││
                    │              │  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘         ││
                    │              │  │         │                │                │                ││
                    │              │  │  ┌──────▼─────────────────▼────────────────▼──────┐         ││
                    │              │  │  │                 Database Subnet                │         ││
                    ▼              │  │  │  ┌────────────┐           ┌────────────┐       │         ││
             ┌──────────┐         │  │  │  │  RDS       │           │ElastiCache │       │         ││
             │    S3    │◀────────┼──┼──│  │ PostgreSQL │           │   Redis    │       │         ││
             │ (Assets) │         │  │  │  │ (pgvector) │           │            │       │         ││
             └──────────┘         │  │  │  └────────────┘           └────────────┘       │         ││
                                  │  │  └─────────────────────────────────────────────────┘         ││
                                  │  └─────────────────────────────────────────────────────────────┘│
                                  └─────────────────────────────────────────────────────────────────┘
```

## Features

### Compute

- **EKS Cluster**: Kubernetes 1.29 with control plane encryption
- **General Purpose Nodes**: m6i/m6a instances for standard workloads
- **GPU Nodes**: g5/g4dn/p4d instances for AI inference
- **Karpenter**: Just-in-time GPU node provisioning with spot support
- **NVIDIA Device Plugin**: GPU resource management

### Networking

- **VPC**: Multi-AZ with public, private, and database subnets
- **NAT Gateway**: Single or per-AZ for cost optimization
- **VPC Endpoints**: S3, ECR, STS, CloudWatch, Secrets Manager
- **Security Groups**: Least-privilege network policies

### Storage

- **PostgreSQL RDS**: pgvector extension for embeddings
- **ElastiCache Redis**: Session state and caching
- **S3 Buckets**: Models, assets, logs, backups
- **CloudFront CDN**: Global distribution for avatar assets

### Security

- **KMS Encryption**: All data encrypted at rest
- **Secrets Manager**: Database and Redis credentials
- **IRSA**: IAM roles for Kubernetes service accounts
- **VPC Flow Logs**: Network traffic auditing

### Observability

- **CloudWatch Alarms**: CPU, memory, storage alerts
- **Container Insights**: EKS pod metrics
- **Performance Insights**: RDS query analysis
- **DCGM Exporter**: GPU metrics for Prometheus

## Prerequisites

- Terraform >= 1.10.0
- AWS CLI configured with appropriate credentials
- kubectl for Kubernetes management
- helm for chart installations

## Usage

### Quick Start (Development)

```bash
cd environments/development
terraform init
terraform plan
terraform apply
```

### Production Deployment

```bash
cd environments/production

# Create terraform.tfvars with production values
cat > terraform.tfvars <<EOF
aws_region                    = "us-east-1"
eks_public_access             = false
cloudfront_aliases            = ["cdn.psyche.oshun.dev"]
cloudfront_acm_certificate_arn = "arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT_ID"
alarm_sns_topic_arns          = ["arn:aws:sns:us-east-1:ACCOUNT:alerts"]
enable_external_dns           = true
hosted_zone_arns              = ["arn:aws:route53:::hostedzone/ZONE_ID"]
EOF

terraform init
terraform plan -out=tfplan
terraform apply tfplan
```

### Connect to EKS Cluster

```bash
# Get the kubectl config command from Terraform output
terraform output -raw kubectl_config | bash

# Verify connection
kubectl get nodes
kubectl get pods -A
```

## Module Structure

```
psyche/
├── versions.tf           # Terraform and provider versions
├── variables.tf          # Input variables
├── locals.tf             # Local values and data sources
├── vpc.tf                # VPC, subnets, NAT, endpoints
├── eks.tf                # EKS cluster, node groups, add-ons
├── karpenter.tf          # Karpenter for GPU auto-scaling
├── rds.tf                # PostgreSQL with pgvector
├── elasticache.tf        # Redis cluster
├── s3.tf                 # Storage buckets
├── cloudfront.tf         # CDN distribution
├── iam.tf                # IRSA roles
├── ecr.tf                # Container registries
├── helm.tf               # Kubernetes add-ons
├── outputs.tf            # Output values
├── README.md             # This file
└── environments/
    ├── development/      # Dev configuration
    ├── staging/          # Staging configuration
    └── production/       # Production configuration
```

## Variables

### Required

| Variable      | Description                           |
| ------------- | ------------------------------------- |
| `environment` | Environment name (dev, staging, prod) |

### Important Optional

| Variable              | Default         | Description                      |
| --------------------- | --------------- | -------------------------------- |
| `project_name`        | `psyche`        | Project name for resource naming |
| `aws_region`          | `us-east-1`     | AWS region                       |
| `vpc_cidr`            | `10.100.0.0/16` | VPC CIDR block                   |
| `az_count`            | `3`             | Number of availability zones     |
| `eks_cluster_version` | `1.29`          | Kubernetes version               |
| `enable_gpu_nodes`    | `true`          | Enable GPU node group            |
| `enable_karpenter`    | `true`          | Enable Karpenter autoscaling     |

See [variables.tf](variables.tf) for the complete list.

## Outputs

| Output                         | Description                   |
| ------------------------------ | ----------------------------- |
| `cluster_name`                 | EKS cluster name              |
| `cluster_endpoint`             | EKS API endpoint              |
| `kubectl_config`               | kubectl configuration command |
| `rds_endpoint`                 | PostgreSQL endpoint           |
| `elasticache_primary_endpoint` | Redis endpoint                |
| `cloudfront_domain_name`       | CDN domain                    |
| `ecr_repository_urls`          | ECR repository URLs           |
| `app_workload_role_arn`        | IAM role for app workloads    |
| `gpu_workload_role_arn`        | IAM role for GPU workloads    |

## Environment Configurations

### Development

- Single NAT gateway
- Spot instances for general nodes
- Minimal GPU capacity (0-2 nodes)
- Single-AZ RDS
- No alarms

### Staging

- Single NAT gateway
- On-demand instances
- Moderate GPU capacity (0-4 nodes)
- Multi-AZ RDS
- Basic alarms

### Production

- NAT per AZ
- On-demand instances only
- Full GPU capacity (1-10 nodes)
- Multi-AZ RDS with read replica
- Full monitoring and alarms

## GPU Instance Types

| Instance     | GPU     | VRAM  | Use Case                        |
| ------------ | ------- | ----- | ------------------------------- |
| g4dn.xlarge  | T4      | 16GB  | Development, small models       |
| g4dn.2xlarge | T4      | 16GB  | Development, batch inference    |
| g5.xlarge    | A10G    | 24GB  | Production, medium models       |
| g5.2xlarge   | A10G    | 24GB  | Production, real-time inference |
| g5.4xlarge   | A10G    | 24GB  | Production, large batch         |
| g5.8xlarge   | A10G    | 24GB  | High-throughput inference       |
| g5.12xlarge  | 4x A10G | 96GB  | Multi-model serving             |
| p4d.24xlarge | 8x A100 | 320GB | Large model training            |

## Cost Optimization

1. **Karpenter Spot Instances**: Configure spot capacity for non-critical GPU
   workloads
2. **Auto-scaling**: Set appropriate min/max nodes
3. **Storage Lifecycle**: Enable S3 lifecycle policies
4. **Single NAT Gateway**: Use in dev/staging

## Security Best Practices

1. **Private EKS API**: Disable public access in production
2. **IRSA**: Use service account roles instead of node roles
3. **Secrets Manager**: Never hardcode credentials
4. **Network Policies**: Implement Kubernetes network policies
5. **Image Scanning**: Enable ECR scan on push

## Troubleshooting

### EKS Cluster Not Ready

```bash
# Check cluster status
aws eks describe-cluster --name psyche-dev --query 'cluster.status'

# Check node groups
aws eks describe-nodegroup --cluster-name psyche-dev --nodegroup-name psyche-dev-general
```

### GPU Nodes Not Scheduling

```bash
# Check NVIDIA device plugin
kubectl get pods -n kube-system -l app.kubernetes.io/name=nvidia-device-plugin

# Check GPU resources
kubectl describe nodes -l nvidia.com/gpu.present=true | grep -A 10 "Allocatable"
```

### Karpenter Not Provisioning

```bash
# Check Karpenter logs
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter

# Check NodePool status
kubectl get nodepools
kubectl get ec2nodeclasses
```

## Maintenance

### Upgrade EKS Version

1. Update `eks_cluster_version` variable
2. Run `terraform plan` to review changes
3. Apply during maintenance window
4. Update node groups sequentially

### Scale GPU Capacity

```bash
# Modify Karpenter limits
kubectl edit nodepool gpu

# Or update Terraform and apply
terraform apply -target=kubectl_manifest.karpenter_node_pool_gpu
```

## Integration with Oshun Platform

This module is part of the Oshun AI Platform monorepo. Related infrastructure:

- **Kubernetes Manifests**: `infra/k8s/psyche/`
- **Docker Configuration**: `docker/psyche/`
- **Application Code**: `libs/psyche/`

## License

Copyright (c) 2024 Oshun Platform. All rights reserved.
