## Phase 91: ML Sovereignty — Continuous Evaluation & Benchmarking

Unified benchmark framework ensuring every model is systematically evaluated
before, during, and after deployment.

---

### 91.1 LLM Evaluation Suite

#### 91.1.1 Standard Benchmarks

- [x] **91.1.1.1** Implement MMLU-Pro benchmark runner (multi-task language
      understanding, augmented options, text+vision)
- [x] **91.1.1.2** Implement HumanEval / BigCodeBench runner (code generation
      with execution verification)
- [x] **91.1.1.3** Implement IFEval runner (instruction following with
      verifiable constraints — automated verification)
- [x] **91.1.1.4** Implement Arena-Hard-Auto evaluation (500 challenging
      prompts, LLM-as-judge scoring)
- [x] **91.1.1.5** Implement GPQA Diamond runner (PhD-level science reasoning)
- [x] **91.1.1.6** Implement MATH-500 benchmark runner (mathematical reasoning)
- [x] **91.1.1.7** Implement LiveBench runner (contamination-free, monthly
      updated prompts across 6 categories)
- [x] **91.1.1.8** Implement SWE-bench Verified runner (real GitHub issue
      resolution)
- [x] **91.1.1.9** Implement WildBench runner (real user prompts from production
      patterns)
- [x] **91.1.1.10** Integrate LightEval (HuggingFace lightweight evaluation —
      unified across all benchmarks)
- [x] **91.1.1.11** Build internal Chatbot Arena (LMSYS-style blind pairwise
      comparison: users see two anonymous model responses, vote for preferred,
      Elo/Bradley-Terry ranking, deployed for internal teams and beta users)
- [x] **91.1.1.12** Implement AAII-methodology evaluation (Artificial Analysis
      Intelligence Index: balanced scoring across Agents, Coding, Scientific
      Reasoning, General categories, hallucination penalty, GDPval-style
      real-world professional tasks)
- [x] **91.1.1.13** Implement Humanity's Last Exam benchmark runner (2,500
      expert-level questions across 100+ subjects, multi-modal, HLE-Rolling
      monthly updates for contamination resistance)
- [x] **91.1.1.14** Implement multi-turn conversation evaluation suite
      (MultiChallenge-style: instruction retention, inference memory, versioned
      editing, self-coherence across turns, MTRAG for multi-turn RAG,
      configurable turn depth)
- [x] **91.1.1.15** Implement agent and tool-use evaluation suite (ToolBench for
      tool retrieval, MCPAgentBench for real-world MCP tasks, multi-step task
      completion scoring, tool selection accuracy, sandboxed execution)

### 91.2 Domain-Specific Evaluation

#### 91.2.1 Per-Domain Benchmarks

- [x] **91.2.1.1** Build RAG evaluation framework using RAGAS methodology
      (faithfulness, relevancy, precision, recall — for Sophia)
- [x] **91.2.1.2** Build embedding quality benchmark using MTEB subset
      (retrieval, classification, clustering, reranking — for embedding models)
- [x] **91.2.1.3** Build NER evaluation suite for Ghanaian entities (precision,
      recall, F1 per entity type — for Veritas)
- [x] **91.2.1.4** Build dialogue consistency evaluation for NPC conversations
      (personality adherence, memory consistency — for Hathor)
- [x] **91.2.1.5** Build TTS quality evaluation suite (MOS prediction, speaker
      similarity, intelligibility — for Psyche and Veritas)
- [x] **91.2.1.6** Build image generation quality evaluation (FID, CLIP score,
      aesthetic score, prompt alignment — for Isis)
- [x] **91.2.1.7** Build 3D generation quality evaluation (geometry accuracy,
      texture quality, PBR correctness — extending Isis benchmarks)
- [x] **91.2.1.8** Build music generation evaluation (audio quality, genre
      accuracy, prompt alignment — for Euterpe)
- [x] **91.2.1.9** Build content moderation precision/recall evaluation (per
      harm category from Kuan Yin taxonomy)
- [x] **91.2.1.10** Build cross-lingual evaluation suite for Ghanaian languages
      (NLU, NLG, translation across Twi/Ewe/Ga/Hausa/Dagbani)

### 91.3 Evaluation Infrastructure

#### 91.3.1 Automated Evaluation Pipeline

- [x] **91.3.1.1** Build unified benchmark runner (execute all registered
      benchmarks against any model endpoint — scheduled daily for production,
      on-demand for checkpoints)
- [x] **91.3.1.2** Build model leaderboard dashboard (per-domain quality over
      time, cross-model comparison, Pareto front on quality vs cost)
- [x] **91.3.1.3** Implement automated regression detection (compare new model
      vs baseline on all benchmarks → alert on statistically significant
      regressions)
- [x] **91.3.1.4** Build human evaluation framework (structured rating tasks,
      random sampling from production, Likert scales, confidence intervals)
- [x] **91.3.1.5** Implement LLM-as-judge evaluation (calibrated judge prompts,
      cross-validate with human ratings)
- [x] **91.3.1.6** Build eval set contamination detector (hash-based exact
      match + embedding-based near match across all training/eval sets)
- [x] **91.3.1.7** Build evaluation report generator (PDF/HTML per model
      version: scores, regressions, comparison charts, recommendations)
- [x] **91.3.1.8** Implement Oshun-Bench (curated challenging prompts from all
      domains, monthly rotation for contamination resistance)
- [x] **91.3.1.9** Build production-traffic eval dataset pipeline (sample and
      annotate real user queries from production → curated eval sets per domain
      with ground truth, quarterly rotation, difficulty stratification)
- [x] **91.3.1.10** Build domain expert eval curation tool (domain experts
      submit challenging prompts + expected responses, quality review workflow,
      Oshun-Bench rotation integration, coverage tracking)
- [x] **91.3.1.11** Build inference performance benchmarking suite (TTFT, TPOT,
      throughput under concurrent load, latency percentiles p50/p95/p99,
      MLPerf-style server scenario, GuideLLM traffic simulation)
- [x] **91.3.1.12** Build cost-performance evaluation dashboard (tokens per
      dollar per model per quantization per GPU, Pareto frontier visualization
      on quality vs cost vs latency, automated regression detection)

Implementation and integration boundaries are documented in
`libs/nous/core/PHASE_91_EVALUATION.md`.

---
