Kubernetes manifests for deploying the Psyche consciousness platform.
Directory Structure#
text
infra/k8s/psyche/
├── base/ # Base Kustomize resources
│ ├── kustomization.yaml # Base kustomization
│ ├── rbac.yaml # RBAC configuration
│ ├── resource-quotas.yaml # Resource quotas
│ ├── limit-ranges.yaml # Container limits
│ ├── network-policies.yaml# Network policies
│ └── priority-classes.yaml# Pod priority classes
├── namespaces/
│ └── namespaces.yaml # Namespace definitions
├── services/
│ ├── reasoning-engine/ # LLM inference service
│ ├── embedding-service/ # Vector embedding service
│ └── tool-executor/ # Tool execution service
├── gpu/
│ ├── node-pool.yaml # GPU node pool config
│ └── karpenter-provisioner.yaml # Karpenter auto-scaling
└── overlays/ # Environment overlays
├── development/
├── staging/
└── production/
Prerequisites#
-
Kubernetes Cluster (EKS, GKE, or AKS) with:
- Kubernetes 1.27+
- NVIDIA GPU Operator installed
- Karpenter or Cluster Autoscaler
-
kubectl configured with cluster access
-
Kustomize (built into kubectl 1.14+)
Namespace Structure#
| Namespace | Purpose |
|---|---|
psyche |
Main application workloads |
psyche-gpu |
GPU-accelerated AI workloads |
psyche-inference |
Model inference serving |
Installation#
1. Install GPU Operator (if not installed)#
bash
# Add NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
# Install GPU Operator
helm install gpu-operator nvidia/gpu-operator \
--namespace gpu-operator \
--create-namespace \
--version v23.9.1
2. Deploy Psyche#
bash
# Development
kubectl apply -k infra/k8s/psyche/overlays/development
# Staging
kubectl apply -k infra/k8s/psyche/overlays/staging
# Production
kubectl apply -k infra/k8s/psyche/overlays/production
3. Verify Deployment#
bash
# Check namespaces
kubectl get namespaces | grep psyche
# Check deployments
kubectl get deployments -n psyche
kubectl get deployments -n psyche-gpu
# Check GPU allocation
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe nodes -l nvidia.com/gpu.present=true | grep -A5 "Allocated resources"
GPU Node Configuration#
Node Labels#
yaml
labels:
node-type: gpu
nvidia.com/gpu.present: 'true'
nvidia.com/gpu.product: NVIDIA-A10G
Node Taints#
yaml
taints:
- key: nvidia.com/gpu
value: 'true'
effect: NoSchedule
GPU Resource Requests#
yaml
resources:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1
Scaling#
Horizontal Pod Autoscaler (HPA)#
Services auto-scale based on:
- GPU utilization (DCGM metrics)
- CPU/Memory utilization
- Custom metrics (request queue length)
Karpenter (Cluster Autoscaler)#
Karpenter provisions GPU nodes based on pending pods:
- Supports spot and on-demand instances
- Multiple GPU instance types (g5, g4dn, p4d)
- Automatic consolidation
Network Policies#
Default policy is deny all with explicit allows:
- DNS resolution
- Intra-namespace communication
- Database access (PostgreSQL, Redis, Qdrant)
- Prometheus scraping
- Cross-namespace (psyche ↔ psyche-gpu)
Resource Quotas#
Production Limits#
| Namespace | CPU | Memory | GPU |
|---|---|---|---|
| psyche | 50 cores | 100Gi | - |
| psyche-gpu | 100 cores | 500Gi | 20 |
| psyche-inference | 80 cores | 400Gi | 16 |
Monitoring#
Metrics exposed at /metrics endpoint:
- Prometheus scrape annotations included
- GPU metrics via DCGM Exporter
- Custom application metrics
Troubleshooting#
Check GPU Availability#
bash
# List GPU nodes
kubectl get nodes -l nvidia.com/gpu.present=true
# Check GPU allocation
kubectl describe node <gpu-node> | grep nvidia
# Check GPU operator pods
kubectl get pods -n gpu-operator
Check Service Health#
bash
# Check pod status
kubectl get pods -n psyche-gpu
# View logs
kubectl logs -n psyche-gpu deployment/reasoning-engine
# Check events
kubectl get events -n psyche-gpu --sort-by='.lastTimestamp'
Network Issues#
bash
# Test connectivity
kubectl run test --rm -it --image=busybox -n psyche -- wget -qO- http://reasoning-engine.psyche-gpu:8010/health
# Check network policies
kubectl get networkpolicies -n psyche -o yaml