Generated by scripts/run_m10_3_evaluations.py report from the committed runs.
Every corpus is synthetic and labelled by its construction; every threshold is
provisional; the launch gate refuses all four. Intervals are 95% Wilson.
item_writing — m10-3-item-writing (run-m10-3-item-writing-20260911T171028Z)#
- task: generation: the unit is a variant request; the right answer is a valid variant
- items 24, answered 24, abstained 0/24 = 0.00 [0.00, 0.14], errored 0/24 = 0.00 [0.00, 0.14]
- validity 21/24 = 0.88 [0.69, 0.96]
- calibration: none (the writer states no confidence in its variants)
- harm: {'key_changed': 1, 'key_given_away': 1, 'operands_lost': 0}
- slices: 13 reported, 13 under 20 items and marked as unable to carry a conclusion
- launch: refused
bias_assist — m10-3-bias-assist (run-m10-3-bias-assist-5e0edfc8947c6d05)#
- task: detection: flag a passage that carries a biased claim, clear one that does not
- items 42, answered 42, abstained 0/42 = 0.00 [0.00, 0.08], errored 0/42 = 0.00 [0.00, 0.08]
- precision 12/18 = 0.67 [0.44, 0.84]
- recall 12/30 = 0.40 [0.25, 0.58]
- false positive rate 6/12 = 0.50 [0.25, 0.75]
- false negative rate 18/30 = 0.60 [0.42, 0.75]
- expected calibration error 0.307 (measured only over predictions that stated a confidence, 18 of 42)
- harm: {'biased_passages_cleared': 18, 'cleared_items': ['bias-gender_trait-paraphrase-1', 'bias-gender_trait-paraphrase-2', 'bias-gender_trait-paraphrase-3', 'bias-cultural_universal-paraphrase-1', 'bias-cultural_universal-paraphrase-2', 'bias-cultural_universal-paraphrase-3', 'bias-language_gate-paraphrase-1', 'bias-language_gate-paraphrase-2', 'bias-language_gate-paraphrase-3', 'bias-normality_default-paraphrase-1', 'bias-normality_default-paraphrase-2', 'bias-normality_default-paraphrase-3', 'bias-sensory_default-paraphrase-1', 'bias-sensory_default-paraphrase-2', 'bias-sensory_default-paraphrase-3', 'bias-representation_limit-paraphrase-1', 'bias-representation_limit-paraphrase-2', 'bias-representation_limit-paraphrase-3']}
- slices: 11 reported, 11 under 20 items and marked as unable to carry a conclusion
- launch: refused
tutor — m10-3-tutor-preview (run-m10-3-tutor-preview-2014a42a94b2af70)#
- task: agreement: each scenario has one expected outcome and code
- items 46, answered 46, abstained 0/46 = 0.00 [0.00, 0.08], errored 0/46 = 0.00 [0.00, 0.08]
- agreement 46/46 = 1.00 [0.92, 1.00]
- calibration: none (the preview engine states no confidence)
- harm: {'disagreements': []}
- slices: 7 reported, 7 under 20 items and marked as unable to carry a conclusion
- launch: refused
integrity_classifier — m10-3-integrity (run-m10-3-integrity-91af9527b3282c81)#
- task: detection: flag a copied, pasted or generated answer; clear the rest
- items 30, answered 30, abstained 0/30 = 0.00 [0.00, 0.11], errored 0/30 = 0.00 [0.00, 0.11]
- precision 4/4 = 1.00 [0.51, 1.00]
- recall 4/22 = 0.18 [0.07, 0.39]
- false positive rate 0/8 = 0.00 [0.00, 0.32]
- false negative rate 18/22 = 0.82 [0.61, 0.93]
- expected calibration error 0.290 (measured only over predictions that stated a confidence, 4 of 30)
- harm: {'learners_wrongly_flagged': 0, 'wrongly_flagged_items': [], 'missed_by_action': {'tutor_copy': 6, 'external_paste': 4, 'generated': 8}}
- slices: 8 reported, 6 under 20 items and marked as unable to carry a conclusion
- launch: refused