Eve library

M10.3 evaluation reports

Evidence

Revision 1

On this page

Generated by scripts/run_m10_3_evaluations.py report from the committed runs. Every corpus is synthetic and labelled by its construction; every threshold is provisional; the launch gate refuses all four. Intervals are 95% Wilson.

item_writing — m10-3-item-writing (run-m10-3-item-writing-20260911T171028Z)#

  • task: generation: the unit is a variant request; the right answer is a valid variant
  • items 24, answered 24, abstained 0/24 = 0.00 [0.00, 0.14], errored 0/24 = 0.00 [0.00, 0.14]
  • validity 21/24 = 0.88 [0.69, 0.96]
  • calibration: none (the writer states no confidence in its variants)
  • harm: {'key_changed': 1, 'key_given_away': 1, 'operands_lost': 0}
  • slices: 13 reported, 13 under 20 items and marked as unable to carry a conclusion
  • launch: refused

bias_assist — m10-3-bias-assist (run-m10-3-bias-assist-5e0edfc8947c6d05)#

  • task: detection: flag a passage that carries a biased claim, clear one that does not
  • items 42, answered 42, abstained 0/42 = 0.00 [0.00, 0.08], errored 0/42 = 0.00 [0.00, 0.08]
  • precision 12/18 = 0.67 [0.44, 0.84]
  • recall 12/30 = 0.40 [0.25, 0.58]
  • false positive rate 6/12 = 0.50 [0.25, 0.75]
  • false negative rate 18/30 = 0.60 [0.42, 0.75]
  • expected calibration error 0.307 (measured only over predictions that stated a confidence, 18 of 42)
  • harm: {'biased_passages_cleared': 18, 'cleared_items': ['bias-gender_trait-paraphrase-1', 'bias-gender_trait-paraphrase-2', 'bias-gender_trait-paraphrase-3', 'bias-cultural_universal-paraphrase-1', 'bias-cultural_universal-paraphrase-2', 'bias-cultural_universal-paraphrase-3', 'bias-language_gate-paraphrase-1', 'bias-language_gate-paraphrase-2', 'bias-language_gate-paraphrase-3', 'bias-normality_default-paraphrase-1', 'bias-normality_default-paraphrase-2', 'bias-normality_default-paraphrase-3', 'bias-sensory_default-paraphrase-1', 'bias-sensory_default-paraphrase-2', 'bias-sensory_default-paraphrase-3', 'bias-representation_limit-paraphrase-1', 'bias-representation_limit-paraphrase-2', 'bias-representation_limit-paraphrase-3']}
  • slices: 11 reported, 11 under 20 items and marked as unable to carry a conclusion
  • launch: refused

tutor — m10-3-tutor-preview (run-m10-3-tutor-preview-2014a42a94b2af70)#

  • task: agreement: each scenario has one expected outcome and code
  • items 46, answered 46, abstained 0/46 = 0.00 [0.00, 0.08], errored 0/46 = 0.00 [0.00, 0.08]
  • agreement 46/46 = 1.00 [0.92, 1.00]
  • calibration: none (the preview engine states no confidence)
  • harm: {'disagreements': []}
  • slices: 7 reported, 7 under 20 items and marked as unable to carry a conclusion
  • launch: refused

integrity_classifier — m10-3-integrity (run-m10-3-integrity-91af9527b3282c81)#

  • task: detection: flag a copied, pasted or generated answer; clear the rest
  • items 30, answered 30, abstained 0/30 = 0.00 [0.00, 0.11], errored 0/30 = 0.00 [0.00, 0.11]
  • precision 4/4 = 1.00 [0.51, 1.00]
  • recall 4/22 = 0.18 [0.07, 0.39]
  • false positive rate 0/8 = 0.00 [0.00, 0.32]
  • false negative rate 18/22 = 0.82 [0.61, 0.93]
  • expected calibration error 0.290 (measured only over predictions that stated a confidence, 4 of 30)
  • harm: {'learners_wrongly_flagged': 0, 'wrongly_flagged_items': [], 'missed_by_action': {'tutor_copy': 6, 'external_paste': 4, 'generated': 8}}
  • slices: 8 reported, 6 under 20 items and marked as unable to carry a conclusion
  • launch: refused