Scope#
WS16-008 publishes a deterministic quality benchmark methodology and
interpretation guide for ISIS benchmark pipelines, scoring models, and
regression assertions.
Research and Existing Implementation Alignment#
The guide is grounded in existing benchmark implementations and published artifacts:
- benchmark runners:
scripts/isis/run-v2-capability-benchmark.mjsscripts/isis/run-v2-ux-interaction-benchmark.mjsscripts/isis/run-v2-control-mode-quality-benchmark.mjs
- benchmark aggregation and gate:
scripts/isis/generate-v2-benchmark-dashboard.mjsscripts/isis/validate-v2-benchmark-regression-gate.mjs
- benchmark artifacts and baseline:
docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.jsondocs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.jsondocs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.jsondocs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.jsondocs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.jsondocs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md
Deliverables#
- Added deterministic methodology guide generator
scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
Generator behavior:
- derives benchmark pipeline methodology from benchmark script contracts;
- derives capability/UX/control scoring formulas and control failure taxonomy;
- derives regression assertion interpretation table from baseline and current benchmark artifacts;
- emits machine-readable and Markdown guide artifacts.
- Added deterministic methodology guide validator
scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
Validator behavior:
- recomputes guide from source scripts and benchmark artifacts;
- verifies JSON/Markdown parity and fails stale output;
- enforces minimum coverage for pipeline steps, assertions, failure classes, and command playbook.
- Published guide artifacts
docs/domains/isis/remediation-v2/ISIS_V2_QUALITY_BENCHMARK_METHODOLOGY_AND_INTERPRETATION_GUIDE.jsondocs/domains/isis/remediation-v2/ISIS_V2_QUALITY_BENCHMARK_METHODOLOGY_AND_INTERPRETATION_GUIDE.md
- Added CI gate
.github/workflows/ci.yml
Added step:
node scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
Verification Commands#
bash
node --check scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
node --check scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/reconcile-v2-doc-claims.mjs
node scripts/isis/validate-v2-doc-claims-reconciliation.mjs
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
Verification Notes#
- Current snapshot captures 5 pipeline methodology steps and 14 regression assertions sourced from the benchmark baseline.
- Current snapshot captures control-mode scoring contract (
workflow=0.55,capability=0.30,modality=0.15, threshold0.75) and 5 failure taxonomy classes.