Eve library

Psyche Kubernetes Infrastructure

Other

Revision 1

On this page

Kubernetes manifests for deploying the Psyche consciousness platform.

Directory Structure#

text
infra/k8s/psyche/
├── base/                    # Base Kustomize resources
│   ├── kustomization.yaml   # Base kustomization
│   ├── rbac.yaml            # RBAC configuration
│   ├── resource-quotas.yaml # Resource quotas
│   ├── limit-ranges.yaml    # Container limits
│   ├── network-policies.yaml# Network policies
│   └── priority-classes.yaml# Pod priority classes
├── namespaces/
│   └── namespaces.yaml      # Namespace definitions
├── services/
│   ├── reasoning-engine/    # LLM inference service
│   ├── embedding-service/   # Vector embedding service
│   └── tool-executor/       # Tool execution service
├── gpu/
│   ├── node-pool.yaml       # GPU node pool config
│   └── karpenter-provisioner.yaml # Karpenter auto-scaling
└── overlays/                # Environment overlays
    ├── development/
    ├── staging/
    └── production/

Prerequisites#

  1. Kubernetes Cluster (EKS, GKE, or AKS) with:

    • Kubernetes 1.27+
    • NVIDIA GPU Operator installed
    • Karpenter or Cluster Autoscaler
  2. kubectl configured with cluster access

  3. Kustomize (built into kubectl 1.14+)

Namespace Structure#

Namespace Purpose
psyche Main application workloads
psyche-gpu GPU-accelerated AI workloads
psyche-inference Model inference serving

Installation#

1. Install GPU Operator (if not installed)#

bash
# Add NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install GPU Operator
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version v23.9.1

2. Deploy Psyche#

bash
# Development
kubectl apply -k infra/k8s/psyche/overlays/development

# Staging
kubectl apply -k infra/k8s/psyche/overlays/staging

# Production
kubectl apply -k infra/k8s/psyche/overlays/production

3. Verify Deployment#

bash
# Check namespaces
kubectl get namespaces | grep psyche

# Check deployments
kubectl get deployments -n psyche
kubectl get deployments -n psyche-gpu

# Check GPU allocation
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe nodes -l nvidia.com/gpu.present=true | grep -A5 "Allocated resources"

GPU Node Configuration#

Node Labels#

yaml
labels:
  node-type: gpu
  nvidia.com/gpu.present: 'true'
  nvidia.com/gpu.product: NVIDIA-A10G

Node Taints#

yaml
taints:
  - key: nvidia.com/gpu
    value: 'true'
    effect: NoSchedule

GPU Resource Requests#

yaml
resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1

Scaling#

Horizontal Pod Autoscaler (HPA)#

Services auto-scale based on:

  • GPU utilization (DCGM metrics)
  • CPU/Memory utilization
  • Custom metrics (request queue length)

Karpenter (Cluster Autoscaler)#

Karpenter provisions GPU nodes based on pending pods:

  • Supports spot and on-demand instances
  • Multiple GPU instance types (g5, g4dn, p4d)
  • Automatic consolidation

Network Policies#

Default policy is deny all with explicit allows:

  • DNS resolution
  • Intra-namespace communication
  • Database access (PostgreSQL, Redis, Qdrant)
  • Prometheus scraping
  • Cross-namespace (psyche ↔ psyche-gpu)

Resource Quotas#

Production Limits#

Namespace CPU Memory GPU
psyche 50 cores 100Gi -
psyche-gpu 100 cores 500Gi 20
psyche-inference 80 cores 400Gi 16

Monitoring#

Metrics exposed at /metrics endpoint:

  • Prometheus scrape annotations included
  • GPU metrics via DCGM Exporter
  • Custom application metrics

Troubleshooting#

Check GPU Availability#

bash
# List GPU nodes
kubectl get nodes -l nvidia.com/gpu.present=true

# Check GPU allocation
kubectl describe node <gpu-node> | grep nvidia

# Check GPU operator pods
kubectl get pods -n gpu-operator

Check Service Health#

bash
# Check pod status
kubectl get pods -n psyche-gpu

# View logs
kubectl logs -n psyche-gpu deployment/reasoning-engine

# Check events
kubectl get events -n psyche-gpu --sort-by='.lastTimestamp'

Network Issues#

bash
# Test connectivity
kubectl run test --rm -it --image=busybox -n psyche -- wget -qO- http://reasoning-engine.psyche-gpu:8010/health

# Check network policies
kubectl get networkpolicies -n psyche -o yaml