Recorded baselines for the Phase 1 verifiers, produced by the eval harness (Task 0.5) over the seed educational gold set (Task 1.10). Every number below is computed by the runner, not hand-written — regenerate it with:
# from libs/metis/verification
npx vitest run src/eval/baselines.spec.ts --reporter=verbose
The runner is src/eval/baselines.ts (computeBaselines); the values are
locked as assertions in src/eval/baselines.spec.ts so a regression in a
verifier is caught.
Gold set#
- File:
src/eval/fixtures/educational-gold.jsonl - 32 labeled items: 16 true facts paired with 16 near-miss corruptions (e.g. "boiling point … 100 degrees" vs "… 50 degrees"), balanced across chemistry, physics, mathematics, astronomy, biology, history, literature, geography.
label= statement is factually correct (should pass);humanScore= human correctness in [0,1]. Provenance: hand-labeled against standard references; intentionally small & balanced for CI calibration.
Headline baseline — factuality verifier#
Configuration: lexical-overlap entailment (+ numeric-contradiction check) over
a true-fact reference corpus, with a grounding-coverage gate (the documented
offline default; lexicalEntailment + lexicalCorpusRetriever +
FactualityVerifier coverage floor 0.5). This is the approximation, not the SOTA
path — it is recorded to quantify what the live embedding/NLI swap (Phase 6.3)
must buy.
| Metric | Value |
|---|---|
| Precision | 0.556 |
| Recall | 0.9375 |
| F1 | 0.698 |
| Accuracy | 0.594 |
| Cohen's κ | 0.1875 |
| Calibration MAE | 0.4066 |
| TP / FP / TN / FN | 15 / 12 / 4 / 1 |
| n | 32 |
Interpretation. This baseline reflects the hardened offline verifier (audit remediation): the lexical entailment now refutes same-subject / different-quantity near-misses ("…50 degrees" vs corpus "…100 degrees"), and the factuality verifier declines to certify content it cannot ground (all-unverifiable → score 0; grounding-coverage floor 0.5). Versus the pre-fix baseline (TP 16 / FP 15 / TN 1 / FN 0, precision 0.516, accuracy 0.531, κ 0.0625): three more near-miss falsehoods are now rejected (FP 15→12, TN 1→4), lifting precision to 0.556, accuracy to 0.594, and κ to 0.1875. The single recall cost (FN 0→1) is one multi-claim TRUE item whose sub-claims the deliberately weak lexical retriever could not fully ground — the coverage gate correctly declines to certify it offline. The remaining ceiling (numeric-only contradiction detection, word-vs-digit blind spots, weak retrieval coverage) is the concrete motivation for the live embedding/NLI retriever + entailment model in Phase 6.3; re-run this harness after that swap to record the SOTA delta.
Note (pre-fix false-pass, now closed): the earlier verifier computed
factScore = supported / (supported+unsupported+contradicted)and returned1when that denominator was0— so a lesson whose every claim was unverifiable (hallucinated, corpus-absent) released as "verified, score 1.0". It now scores0and the coverage gate blocks it.
Per-verifier coverage#
This gold set is a claim-factuality set, so only the factuality verifier is exercised directly. The other verifiers consume different content shapes; their correctness is asserted in their own specs, and each needs a content-shaped gold set before a harness baseline is meaningful.
| Verifier | Exercised here | Where correctness is asserted / what it needs |
|---|---|---|
| factuality | yes | this table; claims/claims.spec.ts |
| faithfulness (TRACe) | no | faithfulness/trace-scorer.spec.ts; needs a context-grounded gold set |
| citation-sufficiency | no | citations/citation-sufficiency.spec.ts; needs a cited-lesson gold set |
| math-correctness | no | stem/math-correctness.spec.ts (e.g. (x+1)^2 = x^2+2x+1 true, = x^2+1 false) |
| code-correctness | no | stem/code-correctness.spec.ts |
| process-verifier | no | stem/process-verifier.spec.ts |
| pedagogical-judge | no | shared @oshun/content-quality-judge suite & pedagogy/judge-verifier.spec.ts; needs an injected panel (model-backed) |
Next baselines to record#
- After Phase 6.3 (live embedding/NLI): re-run factuality; expect precision & κ to rise as near-miss refutation becomes possible.
- Add context-grounded, cited, and worked-solution gold sets to exercise faithfulness, citation-sufficiency, and the STEM verifiers through the same harness.