Eve library

Phase 91 continuous evaluation

Other

Revision 1

On this page

The Phase 91 implementation is intentionally endpoint- and dataset-provider neutral. Public datasets, model credentials, judge endpoints, code sandboxes, PDF rendering, durable storage, and alert transports enter through explicit interfaces; the core never invents benchmark data or silently executes generated code in the application process.

Standard suites#

createStandardBenchmarkRegistry registers MMLU-Pro, HumanEval/BigCodeBench, IFEval, Arena-Hard-Auto, GPQA Diamond, MATH-500, LiveBench, SWE-bench Verified, WildBench, AAII, HLE/HLE-Rolling, multi-turn/MTRAG, and ToolBench/MCPAgentBench. It records canonical dataset references and modality requirements. Code suites require a resource-limited CodeExecutor; Arena-Hard requires a calibrated Judge. runLightEval is the validated LightEval process boundary. BlindArena captures anonymous pairwise votes and bradleyTerryRatings ranks them.

Domain suites#

DOMAIN_EVALUATIONS is the ownership and metric manifest. Concrete metric implementations cover RAGAS, the retrieval/classification/clustering/reranking MTEB subset, typed Ghanaian NER, NPC personality and memory consistency, TTS MOS/speaker/intelligibility, image FID/CLIP/aesthetic and alignment, 3D geometry/texture/PBR, music quality/genre/alignment, per-harm moderation, and per-language NLU/NLG/ChrF for Twi, Ewe, Ga, Hausa, and Dagbani.

Operations and governance#

  • UnifiedBenchmarkRunner executes any registered suite against any ModelEndpoint.
  • EvaluationOrchestrator supports production-daily, checkpoint, and manual runs; persists artifacts, selects a last-known-passing baseline, detects significant regressions, and alerts.
  • judgeCalibration, randomized human sampling, normalized Likert summaries, and confidence intervals connect judge and human evaluation.
  • Exact SHA-256 and embedding-cosine contamination checks cover exact and near matches.
  • HTML and injected-renderer PDF reports are deterministic and escaped.
  • oshunBenchRotation, approved-item validity windows, production redaction, stratified sampling, and independent submitter/reviewer curation implement rotating Oshun-Bench and production sets.
  • benchmarkConcurrentInference captures TTFT, end-to-end p50/p95/p99, token/request throughput, concurrency, and errors. costPerformance and paretoFront support quantization/GPU result dimensions supplied as model metadata.
  • The admin model workspace exposes score history, suite filtering, regressions, and the three-objective quality/cost/latency Pareto front. It has unit and Playwright/axe coverage.

Integration contract#

Production deployments must provide durable EvaluationStore and EvalCurationRepository implementations, an alert sink, authenticated model endpoints, a network-isolated code executor, a calibrated judge, and dataset loaders licensed for the selected benchmark. Local in-memory implementations are supplied for deterministic tests and development, not represented as production persistence.