The Phase 91 implementation is intentionally endpoint- and dataset-provider neutral. Public datasets, model credentials, judge endpoints, code sandboxes, PDF rendering, durable storage, and alert transports enter through explicit interfaces; the core never invents benchmark data or silently executes generated code in the application process.
Standard suites#
createStandardBenchmarkRegistry registers MMLU-Pro, HumanEval/BigCodeBench,
IFEval, Arena-Hard-Auto, GPQA Diamond, MATH-500, LiveBench, SWE-bench Verified,
WildBench, AAII, HLE/HLE-Rolling, multi-turn/MTRAG, and ToolBench/MCPAgentBench.
It records canonical dataset references and modality requirements. Code suites
require a resource-limited CodeExecutor; Arena-Hard requires a calibrated
Judge. runLightEval is the validated LightEval process boundary.
BlindArena captures anonymous pairwise votes and bradleyTerryRatings ranks
them.
Domain suites#
DOMAIN_EVALUATIONS is the ownership and metric manifest. Concrete metric
implementations cover RAGAS, the retrieval/classification/clustering/reranking
MTEB subset, typed Ghanaian NER, NPC personality and memory consistency, TTS
MOS/speaker/intelligibility, image FID/CLIP/aesthetic and alignment, 3D
geometry/texture/PBR, music quality/genre/alignment, per-harm moderation, and
per-language NLU/NLG/ChrF for Twi, Ewe, Ga, Hausa, and Dagbani.
Operations and governance#
UnifiedBenchmarkRunnerexecutes any registered suite against anyModelEndpoint.EvaluationOrchestratorsupports production-daily, checkpoint, and manual runs; persists artifacts, selects a last-known-passing baseline, detects significant regressions, and alerts.judgeCalibration, randomized human sampling, normalized Likert summaries, and confidence intervals connect judge and human evaluation.- Exact SHA-256 and embedding-cosine contamination checks cover exact and near matches.
- HTML and injected-renderer PDF reports are deterministic and escaped.
oshunBenchRotation, approved-item validity windows, production redaction, stratified sampling, and independent submitter/reviewer curation implement rotating Oshun-Bench and production sets.benchmarkConcurrentInferencecaptures TTFT, end-to-end p50/p95/p99, token/request throughput, concurrency, and errors.costPerformanceandparetoFrontsupport quantization/GPU result dimensions supplied as model metadata.- The admin model workspace exposes score history, suite filtering, regressions, and the three-objective quality/cost/latency Pareto front. It has unit and Playwright/axe coverage.
Integration contract#
Production deployments must provide durable EvaluationStore and
EvalCurationRepository implementations, an alert sink, authenticated model
endpoints, a network-isolated code executor, a calibrated judge, and dataset
loaders licensed for the selected benchmark. Local in-memory implementations are
supplied for deterministic tests and development, not represented as production
persistence.