# Psyche Kubernetes Infrastructure

Kubernetes manifests for deploying the Psyche consciousness platform.

## Directory Structure

```
infra/k8s/psyche/
├── base/                    # Base Kustomize resources
│   ├── kustomization.yaml   # Base kustomization
│   ├── rbac.yaml            # RBAC configuration
│   ├── resource-quotas.yaml # Resource quotas
│   ├── limit-ranges.yaml    # Container limits
│   ├── network-policies.yaml# Network policies
│   └── priority-classes.yaml# Pod priority classes
├── namespaces/
│   └── namespaces.yaml      # Namespace definitions
├── services/
│   ├── reasoning-engine/    # LLM inference service
│   ├── embedding-service/   # Vector embedding service
│   └── tool-executor/       # Tool execution service
├── gpu/
│   ├── node-pool.yaml       # GPU node pool config
│   └── karpenter-provisioner.yaml # Karpenter auto-scaling
└── overlays/                # Environment overlays
    ├── development/
    ├── staging/
    └── production/
```

## Prerequisites

1. **Kubernetes Cluster** (EKS, GKE, or AKS) with:
   - Kubernetes 1.27+
   - NVIDIA GPU Operator installed
   - Karpenter or Cluster Autoscaler

2. **kubectl** configured with cluster access

3. **Kustomize** (built into kubectl 1.14+)

## Namespace Structure

| Namespace          | Purpose                      |
| ------------------ | ---------------------------- |
| `psyche`           | Main application workloads   |
| `psyche-gpu`       | GPU-accelerated AI workloads |
| `psyche-inference` | Model inference serving      |

## Installation

### 1. Install GPU Operator (if not installed)

```bash
# Add NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install GPU Operator
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version v23.9.1
```

### 2. Deploy Psyche

```bash
# Development
kubectl apply -k infra/k8s/psyche/overlays/development

# Staging
kubectl apply -k infra/k8s/psyche/overlays/staging

# Production
kubectl apply -k infra/k8s/psyche/overlays/production
```

### 3. Verify Deployment

```bash
# Check namespaces
kubectl get namespaces | grep psyche

# Check deployments
kubectl get deployments -n psyche
kubectl get deployments -n psyche-gpu

# Check GPU allocation
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe nodes -l nvidia.com/gpu.present=true | grep -A5 "Allocated resources"
```

## GPU Node Configuration

### Node Labels

```yaml
labels:
  node-type: gpu
  nvidia.com/gpu.present: 'true'
  nvidia.com/gpu.product: NVIDIA-A10G
```

### Node Taints

```yaml
taints:
  - key: nvidia.com/gpu
    value: 'true'
    effect: NoSchedule
```

### GPU Resource Requests

```yaml
resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1
```

## Scaling

### Horizontal Pod Autoscaler (HPA)

Services auto-scale based on:

- GPU utilization (DCGM metrics)
- CPU/Memory utilization
- Custom metrics (request queue length)

### Karpenter (Cluster Autoscaler)

Karpenter provisions GPU nodes based on pending pods:

- Supports spot and on-demand instances
- Multiple GPU instance types (g5, g4dn, p4d)
- Automatic consolidation

## Network Policies

Default policy is **deny all** with explicit allows:

- DNS resolution
- Intra-namespace communication
- Database access (PostgreSQL, Redis, Qdrant)
- Prometheus scraping
- Cross-namespace (psyche ↔ psyche-gpu)

## Resource Quotas

### Production Limits

| Namespace        | CPU       | Memory | GPU |
| ---------------- | --------- | ------ | --- |
| psyche           | 50 cores  | 100Gi  | -   |
| psyche-gpu       | 100 cores | 500Gi  | 20  |
| psyche-inference | 80 cores  | 400Gi  | 16  |

## Monitoring

Metrics exposed at `/metrics` endpoint:

- Prometheus scrape annotations included
- GPU metrics via DCGM Exporter
- Custom application metrics

## Troubleshooting

### Check GPU Availability

```bash
# List GPU nodes
kubectl get nodes -l nvidia.com/gpu.present=true

# Check GPU allocation
kubectl describe node <gpu-node> | grep nvidia

# Check GPU operator pods
kubectl get pods -n gpu-operator
```

### Check Service Health

```bash
# Check pod status
kubectl get pods -n psyche-gpu

# View logs
kubectl logs -n psyche-gpu deployment/reasoning-engine

# Check events
kubectl get events -n psyche-gpu --sort-by='.lastTimestamp'
```

### Network Issues

```bash
# Test connectivity
kubectl run test --rm -it --image=busybox -n psyche -- wget -qO- http://reasoning-engine.psyche-gpu:8010/health

# Check network policies
kubectl get networkpolicies -n psyche -o yaml
```
