Historical task source. The Eve task board holds current task state.
Phase 95: ML Sovereignty — GPU Infrastructure & Training Clusters#
Multi-cloud GPU orchestration, training cluster management, and cost optimization for running self-hosted model training at scale.
95.1 GPU Cluster Management#
95.1.1 Orchestration & Scheduling#
- 95.1.1.1 Design multi-cloud GPU orchestration layer (RunPod + AWS + GCP as unified pool — abstract provider, expose capacity/cost/availability)
- 95.1.1.2 Build training cluster provisioner (spin up multi-GPU clusters on demand: GPU type × count × hours → provision → configure NCCL → return endpoint)
- 95.1.1.3 Implement spot/preemptible instance manager with checkpointing (60-90% cheaper → checkpoint every N steps → auto-resume on preemption)
- 95.1.1.4 Build GPU memory profiler (predict peak VRAM for model × batch × sequence × method → recommend GPU type before launch)
- 95.1.1.5 Build training cost estimator (predict dollars per run from model size × dataset × epochs × GPU → approve/reject before launch)
- 95.1.1.6 Implement queue-based training scheduler with priority and fairness (domain weights, max concurrent per domain, preemption for urgent)
95.1.2 Monitoring & Optimization#
- 95.1.2.1 Create distributed training launcher (oshun-train-distributed --nodes N --gpus-per-node M → configure NCCL, launch, monitor — one command)
- 95.1.2.2 Build GPU health monitoring dashboard (temperature, utilization, ECC errors, memory, clock speed — alert on anomalies)
- 95.1.2.3 Implement auto-scaling for inference (monitor request queue depth → scale GPU workers up/down → drain before scale-down)
- 95.1.2.4 Build GPU sharing for small models (MIG partitioning, MPS time-sharing for models that don't need full GPU)
- 95.1.2.5 Build training cost dashboard (per-domain dollars per month, per-model per run, per-experiment breakdown, budget alerts)
- 95.1.2.6 Implement GPU utilization alerting (underutilized > 30min → suggest smaller GPU; overloaded → suggest scaling)
95.1.3 Advanced GPU & Network Infrastructure#
- 95.1.3.1 Build RunPod Pod persistent VM lifecycle management (provision/deprovision persistent GPU VMs, manage attached volumes, monitor idle Pods, auto-stop after inactivity, persistent storage management)
- 95.1.3.2 Build NVIDIA DGX/SuperPOD integration for large-scale training runs (DGX B300/GB200 provisioning, NVLink 72-GPU domain configuration, integration with multi-cloud orchestration)
- 95.1.3.3 Build network topology optimization for distributed training (NVLink/NVSwitch fabric configuration, InfiniBand QoS tuning, NCCL topology-aware algorithm selection, RDMA configuration)
- 95.1.3.4 Build checkpoint storage optimization (incremental checkpoints saving only delta parameters, checkpoint compression, deduplication of shared base model weights, tiered storage with auto-archive)
- 95.1.3.5 Build comprehensive training failure recovery (auto-detect NaN loss, OOM, NCCL timeout, hardware failure → auto-resume from last good checkpoint → alert and escalate persistent failures)
- 95.1.3.6 Build carbon footprint tracking for GPU training (track kWh per run, estimate CO2 equivalent based on grid carbon intensity, cumulative sustainability dashboard, per-model carbon budget)
- 95.1.3.7 Build reserved instance optimization analyzer (analyze usage patterns → recommend reserved vs on-demand vs spot allocation → projected savings)
Implementation evidence: libs/nous/training/src/phase-95-capability-catalog.ts
maps every item above to its exported implementation. Behavioral coverage is in
libs/nous/training/src/phase-95-gpu-infrastructure.spec.ts.