# Verifier Eval Baselines — `@metis/verification`

Recorded baselines for the Phase 1 verifiers, produced by the eval harness (Task
0.5) over the seed educational gold set (Task 1.10). Every number below is
**computed by the runner, not hand-written** — regenerate it with:

```bash
# from libs/metis/verification
npx vitest run src/eval/baselines.spec.ts --reporter=verbose
```

The runner is `src/eval/baselines.ts` (`computeBaselines`); the values are
locked as assertions in `src/eval/baselines.spec.ts` so a regression in a
verifier is caught.

## Gold set

- File: `src/eval/fixtures/educational-gold.jsonl`
- 32 labeled items: 16 true facts paired with 16 **near-miss corruptions** (e.g.
  "boiling point … 100 degrees" vs "… 50 degrees"), balanced across chemistry,
  physics, mathematics, astronomy, biology, history, literature, geography.
- `label` = statement is factually correct (should pass); `humanScore` = human
  correctness in [0,1]. Provenance: hand-labeled against standard references;
  intentionally small & balanced for CI calibration.

## Headline baseline — factuality verifier

Configuration: **lexical-overlap entailment (+ numeric-contradiction check) over
a true-fact reference corpus, with a grounding-coverage gate** (the documented
offline default; `lexicalEntailment` + `lexicalCorpusRetriever` +
`FactualityVerifier` coverage floor 0.5). This is the approximation, not the SOTA
path — it is recorded to quantify what the live embedding/NLI swap (Phase 6.3)
must buy.

| Metric            | Value           |
| ----------------- | --------------- |
| Precision         | 0.556           |
| Recall            | 0.9375          |
| F1                | 0.698           |
| Accuracy          | 0.594           |
| Cohen's κ         | 0.1875          |
| Calibration MAE   | 0.4066          |
| TP / FP / TN / FN | 15 / 12 / 4 / 1 |
| n                 | 32              |

**Interpretation.** This baseline reflects the **hardened** offline verifier
(audit remediation): the lexical entailment now **refutes** same-subject /
different-quantity near-misses ("…50 degrees" vs corpus "…100 degrees"), and the
factuality verifier declines to certify content it cannot ground (all-unverifiable
→ score 0; grounding-coverage floor 0.5). Versus the pre-fix baseline
(TP 16 / FP 15 / TN 1 / FN 0, precision 0.516, accuracy 0.531, κ 0.0625): **three
more near-miss falsehoods are now rejected** (FP 15→12, TN 1→4), lifting precision
to 0.556, accuracy to 0.594, and κ to 0.1875. The single recall cost (FN 0→1) is
one multi-claim TRUE item whose sub-claims the **deliberately weak** lexical
retriever could not fully ground — the coverage gate correctly declines to
certify it offline. The remaining ceiling (numeric-only contradiction detection,
word-vs-digit blind spots, weak retrieval coverage) is the concrete motivation for
the live embedding/NLI retriever + entailment model in **Phase 6.3**; re-run this
harness after that swap to record the SOTA delta.

> Note (pre-fix false-pass, now closed): the earlier verifier computed
> `factScore = supported / (supported+unsupported+contradicted)` and returned
> `1` when that denominator was `0` — so a lesson whose every claim was
> *unverifiable* (hallucinated, corpus-absent) released as "verified, score 1.0".
> It now scores `0` and the coverage gate blocks it.

## Per-verifier coverage

This gold set is a **claim-factuality** set, so only the factuality verifier is
exercised directly. The other verifiers consume different content shapes; their
correctness is asserted in their own specs, and each needs a content-shaped gold
set before a harness baseline is meaningful.

| Verifier             | Exercised here | Where correctness is asserted / what it needs                                                                           |
| -------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------- |
| factuality           | yes            | this table; `claims/claims.spec.ts`                                                                                     |
| faithfulness (TRACe) | no             | `faithfulness/trace-scorer.spec.ts`; needs a context-grounded gold set                                                  |
| citation-sufficiency | no             | `citations/citation-sufficiency.spec.ts`; needs a cited-lesson gold set                                                 |
| math-correctness     | no             | `stem/math-correctness.spec.ts` (e.g. `(x+1)^2 = x^2+2x+1` true, `= x^2+1` false)                                       |
| code-correctness     | no             | `stem/code-correctness.spec.ts`                                                                                         |
| process-verifier     | no             | `stem/process-verifier.spec.ts`                                                                                         |
| pedagogical-judge    | no             | shared `@oshun/content-quality-judge` suite & `pedagogy/judge-verifier.spec.ts`; needs an injected panel (model-backed) |

## Next baselines to record

1. After Phase 6.3 (live embedding/NLI): re-run factuality; expect precision & κ
   to rise as near-miss refutation becomes possible.
2. Add context-grounded, cited, and worked-solution gold sets to exercise
   faithfulness, citation-sufficiency, and the STEM verifiers through the same
   harness.
