## Phase 95: ML Sovereignty — GPU Infrastructure & Training Clusters

Multi-cloud GPU orchestration, training cluster management, and cost
optimization for running self-hosted model training at scale.

---

### 95.1 GPU Cluster Management

#### 95.1.1 Orchestration & Scheduling

- [x] **95.1.1.1** Design multi-cloud GPU orchestration layer (RunPod + AWS +
      GCP as unified pool — abstract provider, expose
      capacity/cost/availability)
- [x] **95.1.1.2** Build training cluster provisioner (spin up multi-GPU
      clusters on demand: GPU type × count × hours → provision → configure NCCL
      → return endpoint)
- [x] **95.1.1.3** Implement spot/preemptible instance manager with
      checkpointing (60-90% cheaper → checkpoint every N steps → auto-resume on
      preemption)
- [x] **95.1.1.4** Build GPU memory profiler (predict peak VRAM for model ×
      batch × sequence × method → recommend GPU type before launch)
- [x] **95.1.1.5** Build training cost estimator (predict dollars per run from
      model size × dataset × epochs × GPU → approve/reject before launch)
- [x] **95.1.1.6** Implement queue-based training scheduler with priority and
      fairness (domain weights, max concurrent per domain, preemption for
      urgent)

#### 95.1.2 Monitoring & Optimization

- [x] **95.1.2.1** Create distributed training launcher (oshun-train-distributed
      --nodes N --gpus-per-node M → configure NCCL, launch, monitor — one
      command)
- [x] **95.1.2.2** Build GPU health monitoring dashboard (temperature,
      utilization, ECC errors, memory, clock speed — alert on anomalies)
- [x] **95.1.2.3** Implement auto-scaling for inference (monitor request queue
      depth → scale GPU workers up/down → drain before scale-down)
- [x] **95.1.2.4** Build GPU sharing for small models (MIG partitioning, MPS
      time-sharing for models that don't need full GPU)
- [x] **95.1.2.5** Build training cost dashboard (per-domain dollars per month,
      per-model per run, per-experiment breakdown, budget alerts)
- [x] **95.1.2.6** Implement GPU utilization alerting (underutilized > 30min →
      suggest smaller GPU; overloaded → suggest scaling)

#### 95.1.3 Advanced GPU & Network Infrastructure

- [x] **95.1.3.1** Build RunPod Pod persistent VM lifecycle management
      (provision/deprovision persistent GPU VMs, manage attached volumes,
      monitor idle Pods, auto-stop after inactivity, persistent storage
      management)
- [x] **95.1.3.2** Build NVIDIA DGX/SuperPOD integration for large-scale
      training runs (DGX B300/GB200 provisioning, NVLink 72-GPU domain
      configuration, integration with multi-cloud orchestration)
- [x] **95.1.3.3** Build network topology optimization for distributed training
      (NVLink/NVSwitch fabric configuration, InfiniBand QoS tuning, NCCL
      topology-aware algorithm selection, RDMA configuration)
- [x] **95.1.3.4** Build checkpoint storage optimization (incremental
      checkpoints saving only delta parameters, checkpoint compression,
      deduplication of shared base model weights, tiered storage with
      auto-archive)
- [x] **95.1.3.5** Build comprehensive training failure recovery (auto-detect
      NaN loss, OOM, NCCL timeout, hardware failure → auto-resume from last good
      checkpoint → alert and escalate persistent failures)
- [x] **95.1.3.6** Build carbon footprint tracking for GPU training (track kWh
      per run, estimate CO2 equivalent based on grid carbon intensity,
      cumulative sustainability dashboard, per-model carbon budget)
- [x] **95.1.3.7** Build reserved instance optimization analyzer (analyze usage
      patterns → recommend reserved vs on-demand vs spot allocation → projected
      savings)

---

Implementation evidence: `libs/nous/training/src/phase-95-capability-catalog.ts`
maps every item above to its exported implementation. Behavioral coverage is in
`libs/nous/training/src/phase-95-gpu-infrastructure.spec.ts`.
