# ISIS v2 WS16-008: Quality Benchmark Methodology and Interpretation Guide

## Scope

`WS16-008` publishes a deterministic quality benchmark methodology and
interpretation guide for ISIS benchmark pipelines, scoring models, and
regression assertions.

## Research and Existing Implementation Alignment

The guide is grounded in existing benchmark implementations and published
artifacts:

- benchmark runners:
  - `scripts/isis/run-v2-capability-benchmark.mjs`
  - `scripts/isis/run-v2-ux-interaction-benchmark.mjs`
  - `scripts/isis/run-v2-control-mode-quality-benchmark.mjs`
- benchmark aggregation and gate:
  - `scripts/isis/generate-v2-benchmark-dashboard.mjs`
  - `scripts/isis/validate-v2-benchmark-regression-gate.mjs`
- benchmark artifacts and baseline:
  - `docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json`
  - `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json`
  - `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json`
  - `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json`
  - `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json`
  - `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md`

## Deliverables

1. Added deterministic methodology guide generator

- `scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs`

Generator behavior:

- derives benchmark pipeline methodology from benchmark script contracts;
- derives capability/UX/control scoring formulas and control failure taxonomy;
- derives regression assertion interpretation table from baseline and current
  benchmark artifacts;
- emits machine-readable and Markdown guide artifacts.

2. Added deterministic methodology guide validator

- `scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs`

Validator behavior:

- recomputes guide from source scripts and benchmark artifacts;
- verifies JSON/Markdown parity and fails stale output;
- enforces minimum coverage for pipeline steps, assertions, failure classes, and
  command playbook.

3. Published guide artifacts

- `docs/domains/isis/remediation-v2/ISIS_V2_QUALITY_BENCHMARK_METHODOLOGY_AND_INTERPRETATION_GUIDE.json`
- `docs/domains/isis/remediation-v2/ISIS_V2_QUALITY_BENCHMARK_METHODOLOGY_AND_INTERPRETATION_GUIDE.md`

4. Added CI gate

- `.github/workflows/ci.yml`

Added step:

- `node scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs`

## Verification Commands

```bash
node --check scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
node --check scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/generate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/validate-v2-quality-benchmark-methodology-guide.mjs
node scripts/isis/reconcile-v2-doc-claims.mjs
node scripts/isis/validate-v2-doc-claims-reconciliation.mjs
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

## Verification Notes

- Current snapshot captures 5 pipeline methodology steps and 14 regression
  assertions sourced from the benchmark baseline.
- Current snapshot captures control-mode scoring contract (`workflow=0.55`,
  `capability=0.30`, `modality=0.15`, threshold `0.75`) and 5 failure taxonomy
  classes.
