## Phase 89: ML Sovereignty — Training & Fine-Tuning Infrastructure (SOTA April 2026)

Integrate the latest training frameworks, PEFT methods, alignment techniques,
optimizers, and scaling approaches as of April 2026.

---

### 89.1 Training Frameworks Integration

#### 89.1.1 Framework Setup

- [x] **89.1.1.1** Integrate TRL v1.0 as reference post-training library (75+
      methods: DPO, SimPO, KTO, ORPO, CPO, GRPO, DAPO, RLOO, REINFORCE++, GKD —
      stable API)
- [x] **89.1.1.2** Integrate Unsloth for single-GPU training (12x faster MoE, 7x
      longer context RL, 70% less VRAM, 150+ model support)
- [x] **89.1.1.3** Integrate Axolotl v0.8+ for production-scale training
      (multimodal fine-tuning, QAT, sequence parallelism, GRPO, YAML-driven
      config)
- [x] **89.1.1.4** Integrate torchtune for PyTorch-native training (DoRA
      support, QAT, PPO, PyTorch 2.5 compile for 20-24% speedup)
- [x] **89.1.1.5** Integrate LLaMA-Factory v0.9+ for accessible training
      (LlamaBoard web UI, OFT support, Megatron-LM integration)
- [x] **89.1.1.6** Build unified training launcher CLI (oshun-train --framework
      unsloth/axolotl/torchtune/trl --method lora/dpo/grpo --model --dataset →
      full lifecycle)

### 89.2 Advanced PEFT Methods

#### 89.2.1 Beyond LoRA

- [x] **89.2.1.1** Implement DoRA training support (weight decomposition into
      magnitude + direction — no inference overhead, better convergence)
- [x] **89.2.1.2** Implement VeRA training support (shared frozen random
      matrices + learnable scaling vectors — dramatically fewer params)
- [x] **89.2.1.3** Implement Spectrum layer selection (SNR-based selection of
      highest signal-to-noise layers — compute-efficient)
- [x] **89.2.1.4** Implement QAT pipeline via Axolotl (5x perplexity improvement
      vs post-training quantization — critical for edge)
- [x] **89.2.1.5** Implement AdaLoRA with adaptive rank allocation per layer
      (SVD-based, automatic rank selection)
- [x] **89.2.1.6** Implement multi-LoRA composition training (LoRA Soups
      concatenation + optimal weighting — 43% improvement over merge baselines)

### 89.3 RLVR & Advanced Alignment Training

#### 89.3.1 Reinforcement Learning with Verifiable Rewards

- [x] **89.3.1.1** Implement GRPO training pipeline (Group Relative PO: sample
      8-64 responses → compute group advantages → policy gradient without critic
      — DeepSeek-R1 method)
- [x] **89.3.1.2** Implement DAPO training pipeline (Clip-Higher + Dynamic
      Sampling + Token-level PG Loss + Overlong Reward Shaping — 50% fewer steps
      than GRPO)
- [x] **89.3.1.3** Implement RLVR infrastructure (verifiable reward functions:
      math checker, code sandbox, proof verifier, format validator — replacing
      human labels for reasoning)
- [x] **89.3.1.4** Implement SimPO training (reference-free preference
      optimization using average log probability — +6.4 AlpacaEval 2 vs DPO)
- [x] **89.3.1.5** Implement Process Reward Model training (stepwise reward at
      every reasoning step — ThinkPRM verbalized verification chains)
- [x] **89.3.1.6** Implement self-play training pipeline (SPIRAL: zero-sum game
      self-play → reasoning transfer, +10% on 8 benchmarks)
- [x] **89.3.1.7** Implement online DPO/iterative DPO (generate new responses
      during training → train beyond offline dataset quality)
- [x] **89.3.1.8** Implement RainbowPO training (7 orthogonal DPO improvements
      combined — +28.74% LC Win Rate on AlpacaEval2)
- [x] **89.3.1.9** Implement KTO training for binary feedback (thumbs-up/down
      without pairwise comparisons, models loss aversion)
- [x] **89.3.1.10** Implement multimodal alignment training via MM-DPO (120K
      multimodal preference pairs, critique-based reward model)
- [x] **89.3.1.11** Integrate NeMo-RL as scalable RL training backend (NVIDIA's
      post-training library: Megatron-Core backend, native GRPO + DAPO support,
      multi-GPU/multi-node, NeMo Gym for multi-environment RL, FP8 training,
      HuggingFace compatibility)
- [x] **89.3.1.12** Build standalone reward model training pipeline
      (Bradley-Terry preference model from pairwise data, multi-dimensional
      reward models a la Nemotron, evaluation on RewardBench, deployment as
      reward API for synthetic data filtering and online RLHF)
- [x] **89.3.1.13** Build outcome reward model and process reward model training
      (ORM: final answer correctness scoring, PRM: per-step reasoning
      verification — both needed for RLVR, training from stepwise annotation
      data from 87.4.1.15)

### 89.4 Optimizers & Training Efficiency

#### 89.4.1 Modern Optimizers & Memory Reduction

- [x] **89.4.1.1** Integrate Muon optimizer (Newton-Schulz orthogonalization —
      2x efficiency vs AdamW, 50% cost reduction, 33% memory savings)
- [x] **89.4.1.2** Integrate GaLore 2 for memory-efficient training (gradient
      projection — 82.5% optimizer memory reduction, 63.3% total memory
      reduction)
- [x] **89.4.1.3** Integrate Liger Kernel (fused RMSNorm, RoPE, SwiGLU,
      CrossEntropy — 20% throughput increase, 60% memory reduction)
- [x] **89.4.1.4** Integrate FSDP2 via TorchTitan (per-parameter dim-0 sharding,
      7% lower memory, deterministic, BF16/FP32 mixed precision)
- [x] **89.4.1.5** Integrate DeepSpeed ZeRO++ (4x communication reduction, 50%+
      training time reduction at scale)
- [x] **89.4.1.6** Implement NeMo Megatron Bridge for bidirectional
      HuggingFace/Megatron format conversion
- [x] **89.4.1.7** Build training memory profiler (predict peak VRAM for model +
      batch + sequence + method → right-size GPU)
- [x] **89.4.1.8** Implement curriculum learning for training data scheduling
      (easy-to-hard progression, competence-aware adaptive scheduling,
      length-based online scheduling, configurable difficulty metrics per
      domain)
- [x] **89.4.1.9** Build parallelism strategy auto-configurator (given model
      size + cluster topology + memory budget → select optimal
      DDP/FSDP/FSDP2/DeepSpeed + tensor/pipeline parallelism degrees, with
      hybrid parallelism and communication-computation overlap tuning)
- [x] **89.4.1.10** Build mixed-precision training configuration system
      (BF16/FP32 default with FP8 for Hopper/Blackwell, Transformer Engine
      integration for automatic precision selection, loss scaling, master weight
      management, per-layer precision overrides)

#### 89.4.2 Advanced Training Capabilities

- [x] **89.4.2.1** Build continual pre-training pipeline for domain-adaptive
      pre-training (instruction synthesizer for raw corpora, catastrophic
      forgetting mitigation via replay buffers + elastic weight consolidation,
      configurable domain data mixing, checkpoint evaluation for forgetting
      detection)
- [x] **89.4.2.2** Build mid-training / knowledge injection pipeline (inject new
      domain knowledge between pre-training and fine-tuning: long-form document
      ingestion, knowledge distillation from domain corpora, evaluation-guided
      schedule)
- [x] **89.4.2.3** Build long-context training pipeline (extend model context
      window: YaRN RoPE extension, LongRoPE2 evolutionary search, RoPE base
      frequency adjustment, synthetic long-context data generation, progressive
      extension schedule)

### 89.5 MoE Training & Scaling

#### 89.5.1 Mixture of Experts

- [x] **89.5.1.1** Implement MoE fine-tuning pipeline (fine-tune individual
      experts or routing for Llama 4 Scout/Maverick 16/128-expert architectures)
- [x] **89.5.1.2** Implement MoE expert parallelism via NeMo Megatron Core
      (distribute experts across GPUs with FP8 training)
- [x] **89.5.1.3** Build MoErges/FrankenMoE pipeline (upcycle dense fine-tuned
      models into MoE using mergekit-moe with learned router)
- [x] **89.5.1.4** Implement selective expert fine-tuning (freeze base experts,
      fine-tune only domain-relevant experts)

### 89.6 Model Merging & Composition

#### 89.6.1 Merge Methods

- [x] **89.6.1.1** Integrate mergekit for automated model merging (SLERP, TIES,
      DARE, Task Arithmetic, Passthrough, MoErges — yaml-driven)
- [x] **89.6.1.2** Build automated merge quality evaluation (merge → benchmark →
      compare against sources → accept/reject)
- [x] **89.6.1.3** Implement multi-LoRA routing system (LoRA-Mixer: serial
      attention routing → domain-specific adapters per request)
- [x] **89.6.1.4** Build model merge search (try strategies × weight ratios →
      evaluate → select Pareto-optimal on quality/size)

### 89.7 Multi-Modal Fine-Tuning

#### 89.7.1 VLM & Multi-Modal Training

- [x] **89.7.1.1** Implement multi-modal PEFT for VLM training (vision-language
      LoRA targeting both language and vision encoder layers, visual adapter
      tuning, frozen vision encoder + trainable connector patterns, multi-modal
      QLoRA for memory-efficient VLM fine-tuning)
- [x] **89.7.1.2** Build VLM instruction tuning pipeline (image-text instruction
      following, visual QA fine-tuning, chart/document understanding training,
      multi-image reasoning — supporting Qwen2.5-VL, LLaVA, InternVL
      architectures)

### 89.8 Scaling-Law Discipline & Compute-Optimal Training

No checkpoint should be trained without first proving, on a small-scale IsoFLOP
sweep, that its compute allocation lies on the compute-optimal frontier. These
tasks make scaling-law experimentation a first-class part of every training run
— not a post-hoc rationalisation.

#### 89.8.1 Parametrisation & Transfer

- [x] **89.8.1.1** Implement μTransfer (μP, Yang & Hu) in the training stack —
      maximum-update parametrisation for width and depth, verified by
      shape-invariant optimal learning-rate sweeps on 40M → 400M → 2B width
      scans; optimal HPs found at 40M must transfer zero-shot to 2B (loss within
      1% of independently tuned baseline)
- [x] **89.8.1.2** Implement depth-μP extensions for deep Transformers
      (residual-block scaling) and verify across 12L → 48L → 96L sweeps
- [x] **89.8.1.3** Build scaling-law sweep harness — launch IsoFLOP grids across
      (N_params, D_tokens, batch, LR) parameterised by compute budget C = 6ND,
      fit parametric loss law L(N,D) = E + A/N^α + B/D^β, report compute-optimal
      (N*, D*) per architecture family with bootstrap CIs
- [x] **89.8.1.4** Reproduce Chinchilla-class optimum on a dense Transformer
      (expect D*/N* ≈ 20 tokens/param) and DeepSeek-V2-class MoE optimum
      (separate sparse-active vs total-params law) — publish sovereign
      scaling-law report per architecture

#### 89.8.2 Loss Prediction & Emergence Bench

- [x] **89.8.2.1** Build loss-prediction CI gate — every production training run
      must ship a fitted loss curve from ≤1/16 final compute predicting final
      loss within ±2%; runs that drift outside the band are auto-halted and
      escalated (prevents silent scaling-law violation à la GPT-4-early)
- [x] **89.8.2.2** Build downstream-capability emergence tracker — at each
      Chinchilla decile, probe a fixed eval suite (MMLU, GSM8K, HumanEval, MGSM,
      RewardBench, RULER long-context, SWE-Bench-Verified subset) and record the
      compute threshold at which each capability crosses chance / crosses SOTA,
      producing per-capability emergence curves
- [x] **89.8.2.3** Build data-quality scaling law — pair scaling sweeps with
      dataset-quality variants (web-only, web+code, web+code+math, distilled,
      deduplicated) to quantify quality vs quantity exchange rate
      (token-efficiency multipliers per recipe)
- [x] **89.8.2.4** Build MoE scaling-law experiments — joint scan over (active
      params, total params, n_experts, routing granularity) with RouterLM
      ablations (top-k, expert-choice, soft-MoE) producing compute-optimal
      (N_active, N_total) frontier
- [x] **89.8.2.5** Build multimodal scaling-law experiments extending N, D, and
      modality-mix ratio (text:image:audio:video) — guides Phase 96.1 model
      selection and our own future Oshun-native foundation-model plans

#### 89.8.3 Scaling-Law Governance

- [x] **89.8.3.1** Store every scaling-law experiment (configs, checkpoints,
      loss traces, fitted parameters) in the Phase 92 experiment store with
      immutable hashes — make scaling-law data reusable across domain training
      recipes
- [x] **89.8.3.2** Require every Phase 89 training recipe to declare target (N,
      D, C) with a pointer to the scaling-law study that justifies it — recipe
      linter fails CI if the declaration is missing or inconsistent with
      published laws
- [x] **89.8.3.3** Publish quarterly sovereign scaling-law bulletin (open
      artefact) so Oshun-internal recipes stay calibrated against the evolving
      external frontier (Chinchilla → Hoffmann → Llama-3 / DeepSeek V3 refits)

---

## Completion evidence

All 60 locally actionable items are implemented in
`libs/nous/training/src/phase-89-*.ts` and the `oshun-train` CLI.
`phase-89-capability-catalog.ts` maintains an exact one-to-one mapping from
every checklist ID to its implementation and automated verification; its test
reads this checklist and fails on missing, duplicate, or unexpected evidence.

Verification: `nx test nous-training`, `nx lint nous-training`,
`nx build nous-training`, `nx typecheck nous-training`, and the hash-resolving
`nx recipe-lint nous-training` CI gate.
