Eve library

Verifier Eval Baselines — `@metis/verification`

Other

Revision 1

On this page

Recorded baselines for the Phase 1 verifiers, produced by the eval harness (Task 0.5) over the seed educational gold set (Task 1.10). Every number below is computed by the runner, not hand-written — regenerate it with:

bash
# from libs/metis/verification
npx vitest run src/eval/baselines.spec.ts --reporter=verbose

The runner is src/eval/baselines.ts (computeBaselines); the values are locked as assertions in src/eval/baselines.spec.ts so a regression in a verifier is caught.

Gold set#

  • File: src/eval/fixtures/educational-gold.jsonl
  • 32 labeled items: 16 true facts paired with 16 near-miss corruptions (e.g. "boiling point … 100 degrees" vs "… 50 degrees"), balanced across chemistry, physics, mathematics, astronomy, biology, history, literature, geography.
  • label = statement is factually correct (should pass); humanScore = human correctness in [0,1]. Provenance: hand-labeled against standard references; intentionally small & balanced for CI calibration.

Headline baseline — factuality verifier#

Configuration: lexical-overlap entailment (+ numeric-contradiction check) over a true-fact reference corpus, with a grounding-coverage gate (the documented offline default; lexicalEntailment + lexicalCorpusRetriever + FactualityVerifier coverage floor 0.5). This is the approximation, not the SOTA path — it is recorded to quantify what the live embedding/NLI swap (Phase 6.3) must buy.

Metric Value
Precision 0.556
Recall 0.9375
F1 0.698
Accuracy 0.594
Cohen's κ 0.1875
Calibration MAE 0.4066
TP / FP / TN / FN 15 / 12 / 4 / 1
n 32

Interpretation. This baseline reflects the hardened offline verifier (audit remediation): the lexical entailment now refutes same-subject / different-quantity near-misses ("…50 degrees" vs corpus "…100 degrees"), and the factuality verifier declines to certify content it cannot ground (all-unverifiable → score 0; grounding-coverage floor 0.5). Versus the pre-fix baseline (TP 16 / FP 15 / TN 1 / FN 0, precision 0.516, accuracy 0.531, κ 0.0625): three more near-miss falsehoods are now rejected (FP 15→12, TN 1→4), lifting precision to 0.556, accuracy to 0.594, and κ to 0.1875. The single recall cost (FN 0→1) is one multi-claim TRUE item whose sub-claims the deliberately weak lexical retriever could not fully ground — the coverage gate correctly declines to certify it offline. The remaining ceiling (numeric-only contradiction detection, word-vs-digit blind spots, weak retrieval coverage) is the concrete motivation for the live embedding/NLI retriever + entailment model in Phase 6.3; re-run this harness after that swap to record the SOTA delta.

Note (pre-fix false-pass, now closed): the earlier verifier computed factScore = supported / (supported+unsupported+contradicted) and returned 1 when that denominator was 0 — so a lesson whose every claim was unverifiable (hallucinated, corpus-absent) released as "verified, score 1.0". It now scores 0 and the coverage gate blocks it.

Per-verifier coverage#

This gold set is a claim-factuality set, so only the factuality verifier is exercised directly. The other verifiers consume different content shapes; their correctness is asserted in their own specs, and each needs a content-shaped gold set before a harness baseline is meaningful.

Verifier Exercised here Where correctness is asserted / what it needs
factuality yes this table; claims/claims.spec.ts
faithfulness (TRACe) no faithfulness/trace-scorer.spec.ts; needs a context-grounded gold set
citation-sufficiency no citations/citation-sufficiency.spec.ts; needs a cited-lesson gold set
math-correctness no stem/math-correctness.spec.ts (e.g. (x+1)^2 = x^2+2x+1 true, = x^2+1 false)
code-correctness no stem/code-correctness.spec.ts
process-verifier no stem/process-verifier.spec.ts
pedagogical-judge no shared @oshun/content-quality-judge suite & pedagogy/judge-verifier.spec.ts; needs an injected panel (model-backed)

Next baselines to record#

  1. After Phase 6.3 (live embedding/NLI): re-run factuality; expect precision & κ to rise as near-miss refutation becomes possible.
  2. Add context-grounded, cited, and worked-solution gold sets to exercise faithfulness, citation-sufficiency, and the STEM verifiers through the same harness.