Eve library

ISIS v2 WS16-008: Quality Benchmark Methodology and Interpretation Guide

Guide

Revision 1

On this page

Scope#

WS16-008 publishes a deterministic quality benchmark methodology and interpretation guide for ISIS benchmark pipelines, scoring models, and regression assertions.

Research and Existing Implementation Alignment#

The guide is grounded in existing benchmark implementations and published artifacts:

  • benchmark runners:
    • scripts/isis/run-v2-capability-benchmark.mjs
    • scripts/isis/run-v2-ux-interaction-benchmark.mjs
    • scripts/isis/run-v2-control-mode-quality-benchmark.mjs
  • benchmark aggregation and gate:
    • scripts/isis/generate-v2-benchmark-dashboard.mjs
    • scripts/isis/validate-v2-benchmark-regression-gate.mjs
  • benchmark artifacts and baseline:
    • docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json
    • docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json
    • docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json
    • docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json
    • docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json
    • docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md

Deliverables#

  1. Added deterministic methodology guide generator
  • scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs

Generator behavior:

  • derives benchmark pipeline methodology from benchmark script contracts;
  • derives capability/UX/control scoring formulas and control failure taxonomy;
  • derives regression assertion interpretation table from baseline and current benchmark artifacts;
  • emits machine-readable and Markdown guide artifacts.
  1. Added deterministic methodology guide validator
  • scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs

Validator behavior:

  • recomputes guide from source scripts and benchmark artifacts;
  • verifies JSON/Markdown parity and fails stale output;
  • enforces minimum coverage for pipeline steps, assertions, failure classes, and command playbook.
  1. Published guide artifacts
  • docs/domains/isis/remediation-v2/ISIS_V2_QUALITY_BENCHMARK_METHODOLOGY_AND_INTERPRETATION_GUIDE.json
  • docs/domains/isis/remediation-v2/ISIS_V2_QUALITY_BENCHMARK_METHODOLOGY_AND_INTERPRETATION_GUIDE.md
  1. Added CI gate
  • .github/workflows/ci.yml

Added step:

  • node scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs

Verification Commands#

bash
node --check scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
node --check scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/reconcile-v2-doc-claims.mjs
node scripts/isis/validate-v2-doc-claims-reconciliation.mjs
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Verification Notes#

  • Current snapshot captures 5 pipeline methodology steps and 14 regression assertions sourced from the benchmark baseline.
  • Current snapshot captures control-mode scoring contract (workflow=0.55, capability=0.30, modality=0.15, threshold 0.75) and 5 failure taxonomy classes.