Historical task source. The Eve task board holds current task state.
Phase 91: ML Sovereignty — Continuous Evaluation & Benchmarking#
Unified benchmark framework ensuring every model is systematically evaluated before, during, and after deployment.
91.1 LLM Evaluation Suite#
91.1.1 Standard Benchmarks#
- 91.1.1.1 Implement MMLU-Pro benchmark runner (multi-task language understanding, augmented options, text+vision)
- 91.1.1.2 Implement HumanEval / BigCodeBench runner (code generation with execution verification)
- 91.1.1.3 Implement IFEval runner (instruction following with verifiable constraints — automated verification)
- 91.1.1.4 Implement Arena-Hard-Auto evaluation (500 challenging prompts, LLM-as-judge scoring)
- 91.1.1.5 Implement GPQA Diamond runner (PhD-level science reasoning)
- 91.1.1.6 Implement MATH-500 benchmark runner (mathematical reasoning)
- 91.1.1.7 Implement LiveBench runner (contamination-free, monthly updated prompts across 6 categories)
- 91.1.1.8 Implement SWE-bench Verified runner (real GitHub issue resolution)
- 91.1.1.9 Implement WildBench runner (real user prompts from production patterns)
- 91.1.1.10 Integrate LightEval (HuggingFace lightweight evaluation — unified across all benchmarks)
- 91.1.1.11 Build internal Chatbot Arena (LMSYS-style blind pairwise comparison: users see two anonymous model responses, vote for preferred, Elo/Bradley-Terry ranking, deployed for internal teams and beta users)
- 91.1.1.12 Implement AAII-methodology evaluation (Artificial Analysis Intelligence Index: balanced scoring across Agents, Coding, Scientific Reasoning, General categories, hallucination penalty, GDPval-style real-world professional tasks)
- 91.1.1.13 Implement Humanity's Last Exam benchmark runner (2,500 expert-level questions across 100+ subjects, multi-modal, HLE-Rolling monthly updates for contamination resistance)
- 91.1.1.14 Implement multi-turn conversation evaluation suite (MultiChallenge-style: instruction retention, inference memory, versioned editing, self-coherence across turns, MTRAG for multi-turn RAG, configurable turn depth)
- 91.1.1.15 Implement agent and tool-use evaluation suite (ToolBench for tool retrieval, MCPAgentBench for real-world MCP tasks, multi-step task completion scoring, tool selection accuracy, sandboxed execution)
91.2 Domain-Specific Evaluation#
91.2.1 Per-Domain Benchmarks#
- 91.2.1.1 Build RAG evaluation framework using RAGAS methodology (faithfulness, relevancy, precision, recall — for Sophia)
- 91.2.1.2 Build embedding quality benchmark using MTEB subset (retrieval, classification, clustering, reranking — for embedding models)
- 91.2.1.3 Build NER evaluation suite for Ghanaian entities (precision, recall, F1 per entity type — for Veritas)
- 91.2.1.4 Build dialogue consistency evaluation for NPC conversations (personality adherence, memory consistency — for Hathor)
- 91.2.1.5 Build TTS quality evaluation suite (MOS prediction, speaker similarity, intelligibility — for Psyche and Veritas)
- 91.2.1.6 Build image generation quality evaluation (FID, CLIP score, aesthetic score, prompt alignment — for Isis)
- 91.2.1.7 Build 3D generation quality evaluation (geometry accuracy, texture quality, PBR correctness — extending Isis benchmarks)
- 91.2.1.8 Build music generation evaluation (audio quality, genre accuracy, prompt alignment — for Euterpe)
- 91.2.1.9 Build content moderation precision/recall evaluation (per harm category from Kuan Yin taxonomy)
- 91.2.1.10 Build cross-lingual evaluation suite for Ghanaian languages (NLU, NLG, translation across Twi/Ewe/Ga/Hausa/Dagbani)
91.3 Evaluation Infrastructure#
91.3.1 Automated Evaluation Pipeline#
- 91.3.1.1 Build unified benchmark runner (execute all registered benchmarks against any model endpoint — scheduled daily for production, on-demand for checkpoints)
- 91.3.1.2 Build model leaderboard dashboard (per-domain quality over time, cross-model comparison, Pareto front on quality vs cost)
- 91.3.1.3 Implement automated regression detection (compare new model vs baseline on all benchmarks → alert on statistically significant regressions)
- 91.3.1.4 Build human evaluation framework (structured rating tasks, random sampling from production, Likert scales, confidence intervals)
- 91.3.1.5 Implement LLM-as-judge evaluation (calibrated judge prompts, cross-validate with human ratings)
- 91.3.1.6 Build eval set contamination detector (hash-based exact match + embedding-based near match across all training/eval sets)
- 91.3.1.7 Build evaluation report generator (PDF/HTML per model version: scores, regressions, comparison charts, recommendations)
- 91.3.1.8 Implement Oshun-Bench (curated challenging prompts from all domains, monthly rotation for contamination resistance)
- 91.3.1.9 Build production-traffic eval dataset pipeline (sample and annotate real user queries from production → curated eval sets per domain with ground truth, quarterly rotation, difficulty stratification)
- 91.3.1.10 Build domain expert eval curation tool (domain experts submit challenging prompts + expected responses, quality review workflow, Oshun-Bench rotation integration, coverage tracking)
- 91.3.1.11 Build inference performance benchmarking suite (TTFT, TPOT, throughput under concurrent load, latency percentiles p50/p95/p99, MLPerf-style server scenario, GuideLLM traffic simulation)
- 91.3.1.12 Build cost-performance evaluation dashboard (tokens per dollar per model per quantization per GPU, Pareto frontier visualization on quality vs cost vs latency, automated regression detection)
Implementation and integration boundaries are documented in
libs/nous/core/PHASE_91_EVALUATION.md.