# ISIS v2 Quality Benchmark Methodology and Interpretation Guide

Generated: 2026-03-06T11:20:13.891Z Scope: `WS16-008`

## Summary

- Benchmark pipeline steps: 5
- Regression assertions: 14
- Failing assertions: 0
- Control failure classes: 5
- Methodology formulas captured: 6

## Benchmark Pipeline Methodology

| Step                            | Checklist  | Script                                                   | Inputs                                                                                                                                                                                                                                                                                                                                                                                                                                    | Outputs                                                                                                                                                      |
| ------------------------------- | ---------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Capability benchmark            | `WS01-007` | `scripts/isis/run-v2-capability-benchmark.mjs`           | `docs/domains/isis/remediation-v2/ISIS_V2_COMFY_CAPABILITY_MATRIX.json`, `libs/isis/workflows/src/workflows`                                                                                                                                                                                                                                                                                                                              | `docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.md`                     |
| UX interaction benchmark        | `WS01-008` | `scripts/isis/run-v2-ux-interaction-benchmark.mjs`       | `apps/isis/web/src/pages/WorkflowEditorPage.tsx`, `apps/isis/web/src/pages/WorkflowTemplatesPage.tsx`, `apps/isis/web/src/pages/WorkflowsPage.tsx`, `docs/domains/isis/remediation-v2/ISIS_V2_ASSET_TAXONOMY.json`, `docs/domains/isis/remediation-v2/ISIS_V2_OUTPUT_PACKAGE_TAXONOMY.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json`, `libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json`    | `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.md`             |
| Control-mode quality benchmark  | `WS05-009` | `scripts/isis/run-v2-control-mode-quality-benchmark.mjs` | `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_TAXONOMY.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json`, `libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json`                                                                                                                                                     | `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.md` |
| Benchmark dashboard aggregation | `WS01-009` | `scripts/isis/generate-v2-benchmark-dashboard.mjs`       | `docs/domains/isis/remediation-v2/ISIS_V2_ASSET_TAXONOMY.json`, `docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json`, `libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json` | `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json`, `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.md`                       |
| Benchmark regression gate       | `WS01-010` | `scripts/isis/validate-v2-benchmark-regression-gate.mjs` | `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json`, `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json`, `docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json`, `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json`                                                        | `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md`                                                                                |

## Metric Formulas

### Capability Benchmark

- nodeTargetPassRate = (requiredNodeTargets - failedRequiredNodeTargets) /
  requiredNodeTargets
- workflowCapabilityPassRate = (requiredWorkflowCapabilities -
  missingRequiredCapabilities) / requiredWorkflowCapabilities

### UX Benchmark

- requiredCapabilityPassRate = passedRequiredCapabilities / requiredCapabilities
- workflowPassRate = workflowsPassed / workflowCoverage
- taxonomyCategoryCoverage = matchedTaxonomyCategories / taxonomyCategories

### Control-Mode Quality Scoring

- Pass threshold: `0.75`
- Weights: workflowPassRate=`0.55`, capabilityPassRate=`0.3`,
  modalityCoverage=`0.15`
- Formula: qualityScore = workflowPassRate*workflowWeight +
  capabilityPassRate*capabilityWeight + modalityCoverage\*modalityWeight

## Regression Assertion Interpretation

- Baseline description: Minimum acceptable benchmark levels. Pass-rate metrics
  must not decrease; missing-gap metrics must not increase.

| Assertion                                                 | Metric                                        |             Actual | Rule |          Threshold |    Margin | Result | Interpretation                                      |
| --------------------------------------------------------- | --------------------------------------------- | -----------------: | ---- | -----------------: | --------: | ------ | --------------------------------------------------- |
| Capability workflow pass rate non-regression              | `capability.workflowCapabilityPassRate`       |                  1 | >=   | 0.7777777777777778 |  0.222222 | PASS   | Within guardrail threshold.                         |
| Capability node-class pass rate non-regression            | `capability.nodeTargetPassRate`               | 0.9166666666666666 | >=   | 0.8333333333333334 |  0.083333 | PASS   | Within guardrail threshold.                         |
| Missing required workflow capabilities must not increase  | `capability.missingRequiredCapabilitiesCount` |                  0 | <=   |                  4 |  4.000000 | PASS   | Within guardrail threshold.                         |
| Failed required node targets must not increase            | `capability.failedRequiredNodeTargetsCount`   |                  1 | <=   |                  2 |  1.000000 | PASS   | Within guardrail threshold.                         |
| UX required capability pass rate non-regression           | `ux.requiredCapabilityPassRate`               |                  1 | >=   |                  1 |  0.000000 | PASS   | At threshold. Treat as fragile and monitor closely. |
| UX workflow pass rate non-regression                      | `ux.workflowPassRate`                         |                  1 | >=   |                  1 |  0.000000 | PASS   | At threshold. Treat as fragile and monitor closely. |
| UX passed workflow count non-regression                   | `ux.workflowsPassed`                          |                 48 | >=   |                 48 |  0.000000 | PASS   | At threshold. Treat as fragile and monitor closely. |
| Missing required UX capabilities must not increase        | `ux.missingRequiredCapabilityCount`           |                  0 | <=   |                  0 |  0.000000 | PASS   | At threshold. Treat as fragile and monitor closely. |
| Missing UX capability bucket count must not increase      | `ux.missingCapabilityBuckets`                 |                  0 | <=   |                  0 |  0.000000 | PASS   | At threshold. Treat as fragile and monitor closely. |
| Taxonomy category coverage non-regression                 | `ux.taxonomyCategoryCoverage`                 |                  1 | >=   |                  1 |  0.000000 | PASS   | At threshold. Treat as fragile and monitor closely. |
| Control-mode average quality score non-regression         | `control.averageQualityScore`                 |           0.972039 | >=   |           0.251944 |  0.720095 | PASS   | Within guardrail threshold.                         |
| Failing control-mode count must not increase              | `control.failingControlCount`                 |                  7 | <=   |                 27 | 20.000000 | PASS   | Within guardrail threshold.                         |
| High-severity control failures must not increase          | `control.highSeverityFailureCount`            |                  0 | <=   |                 27 | 27.000000 | PASS   | Within guardrail threshold.                         |
| Control modes with no workflow coverage must not increase | `control.noWorkflowCoverageCount`             |                  0 | <=   |                  5 |  5.000000 | PASS   | Within guardrail threshold.                         |

## Control Failure Taxonomy

| Class                   | Severity | Description                                                                                       |
| ----------------------- | -------- | ------------------------------------------------------------------------------------------------- |
| `benchmark-score-gap`   | medium   | Composite benchmark score is below pass threshold (0.75).                                         |
| `modality-coverage-gap` | medium   | Observed workflow modality coverage for the control does not match declared supported modalities. |
| `no-workflow-coverage`  | critical | Control modality is defined in taxonomy but not used by any workflow.                             |
| `ux-capability-gap`     | high     | Control-specific UX capability mappings are missing or failing benchmark probes.                  |
| `workflow-quality-gap`  | high     | One or more workflows using this control fail required UX quality coverage checks.                |

## Command Playbook

- `node scripts/isis/run-v2-capability-benchmark.mjs`
- `node scripts/isis/run-v2-ux-interaction-benchmark.mjs`
- `node scripts/isis/run-v2-control-mode-quality-benchmark.mjs`
- `node scripts/isis/generate-v2-benchmark-dashboard.mjs`
- `node scripts/isis/validate-v2-benchmark-regression-gate.mjs`

## Source References

- `scripts`: `scripts/isis/run-v2-capability-benchmark.mjs`,
  `scripts/isis/run-v2-ux-interaction-benchmark.mjs`,
  `scripts/isis/run-v2-control-mode-quality-benchmark.mjs`,
  `scripts/isis/generate-v2-benchmark-dashboard.mjs`,
  `scripts/isis/validate-v2-benchmark-regression-gate.mjs`
- `benchmarkArtifacts`:
  `docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json`,
  `docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json`,
  `docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json`,
  `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json`,
  `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json`,
  `docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md`
