Eve library
ISIS v2 Quality Benchmark Methodology and Interpretation Guide
Guide
Revision 1
On this page
Generated: 2026-03-06T11:20:13.891Z Scope: WS16-008
Summary#
- Benchmark pipeline steps: 5
- Regression assertions: 14
- Failing assertions: 0
- Control failure classes: 5
- Methodology formulas captured: 6
Benchmark Pipeline Methodology#
| Step |
Checklist |
Script |
Inputs |
Outputs |
| Capability benchmark |
WS01-007 |
scripts/isis/run-v2-capability-benchmark.mjs |
docs/domains/isis/remediation-v2/ISIS_V2_COMFY_CAPABILITY_MATRIX.json, libs/isis/workflows/src/workflows |
docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.md |
| UX interaction benchmark |
WS01-008 |
scripts/isis/run-v2-ux-interaction-benchmark.mjs |
apps/isis/web/src/pages/WorkflowEditorPage.tsx, apps/isis/web/src/pages/WorkflowTemplatesPage.tsx, apps/isis/web/src/pages/WorkflowsPage.tsx, docs/domains/isis/remediation-v2/ISIS_V2_ASSET_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_OUTPUT_PACKAGE_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json, libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json |
docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.md |
| Control-mode quality benchmark |
WS05-009 |
scripts/isis/run-v2-control-mode-quality-benchmark.mjs |
docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json, libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json |
docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.md |
| Benchmark dashboard aggregation |
WS01-009 |
scripts/isis/generate-v2-benchmark-dashboard.mjs |
docs/domains/isis/remediation-v2/ISIS_V2_ASSET_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json, libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json |
docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.md |
| Benchmark regression gate |
WS01-010 |
scripts/isis/validate-v2-benchmark-regression-gate.mjs |
docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json, docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json |
docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md |
Capability Benchmark#
- nodeTargetPassRate = (requiredNodeTargets - failedRequiredNodeTargets) /
requiredNodeTargets
- workflowCapabilityPassRate = (requiredWorkflowCapabilities -
missingRequiredCapabilities) / requiredWorkflowCapabilities
UX Benchmark#
- requiredCapabilityPassRate = passedRequiredCapabilities / requiredCapabilities
- workflowPassRate = workflowsPassed / workflowCoverage
- taxonomyCategoryCoverage = matchedTaxonomyCategories / taxonomyCategories
Control-Mode Quality Scoring#
- Pass threshold:
0.75
- Weights: workflowPassRate=
0.55, capabilityPassRate=0.3,
modalityCoverage=0.15
- Formula: qualityScore = workflowPassRateworkflowWeight +
capabilityPassRatecapabilityWeight + modalityCoverage*modalityWeight
Regression Assertion Interpretation#
- Baseline description: Minimum acceptable benchmark levels. Pass-rate metrics
must not decrease; missing-gap metrics must not increase.
| Assertion |
Metric |
Actual |
Rule |
Threshold |
Margin |
Result |
Interpretation |
| Capability workflow pass rate non-regression |
capability.workflowCapabilityPassRate |
1 |
>= |
0.7777777777777778 |
0.222222 |
PASS |
Within guardrail threshold. |
| Capability node-class pass rate non-regression |
capability.nodeTargetPassRate |
0.9166666666666666 |
>= |
0.8333333333333334 |
0.083333 |
PASS |
Within guardrail threshold. |
| Missing required workflow capabilities must not increase |
capability.missingRequiredCapabilitiesCount |
0 |
<= |
4 |
4.000000 |
PASS |
Within guardrail threshold. |
| Failed required node targets must not increase |
capability.failedRequiredNodeTargetsCount |
1 |
<= |
2 |
1.000000 |
PASS |
Within guardrail threshold. |
| UX required capability pass rate non-regression |
ux.requiredCapabilityPassRate |
1 |
>= |
1 |
0.000000 |
PASS |
At threshold. Treat as fragile and monitor closely. |
| UX workflow pass rate non-regression |
ux.workflowPassRate |
1 |
>= |
1 |
0.000000 |
PASS |
At threshold. Treat as fragile and monitor closely. |
| UX passed workflow count non-regression |
ux.workflowsPassed |
48 |
>= |
48 |
0.000000 |
PASS |
At threshold. Treat as fragile and monitor closely. |
| Missing required UX capabilities must not increase |
ux.missingRequiredCapabilityCount |
0 |
<= |
0 |
0.000000 |
PASS |
At threshold. Treat as fragile and monitor closely. |
| Missing UX capability bucket count must not increase |
ux.missingCapabilityBuckets |
0 |
<= |
0 |
0.000000 |
PASS |
At threshold. Treat as fragile and monitor closely. |
| Taxonomy category coverage non-regression |
ux.taxonomyCategoryCoverage |
1 |
>= |
1 |
0.000000 |
PASS |
At threshold. Treat as fragile and monitor closely. |
| Control-mode average quality score non-regression |
control.averageQualityScore |
0.972039 |
>= |
0.251944 |
0.720095 |
PASS |
Within guardrail threshold. |
| Failing control-mode count must not increase |
control.failingControlCount |
7 |
<= |
27 |
20.000000 |
PASS |
Within guardrail threshold. |
| High-severity control failures must not increase |
control.highSeverityFailureCount |
0 |
<= |
27 |
27.000000 |
PASS |
Within guardrail threshold. |
| Control modes with no workflow coverage must not increase |
control.noWorkflowCoverageCount |
0 |
<= |
5 |
5.000000 |
PASS |
Within guardrail threshold. |
Control Failure Taxonomy#
| Class |
Severity |
Description |
benchmark-score-gap |
medium |
Composite benchmark score is below pass threshold (0.75). |
modality-coverage-gap |
medium |
Observed workflow modality coverage for the control does not match declared supported modalities. |
no-workflow-coverage |
critical |
Control modality is defined in taxonomy but not used by any workflow. |
ux-capability-gap |
high |
Control-specific UX capability mappings are missing or failing benchmark probes. |
workflow-quality-gap |
high |
One or more workflows using this control fail required UX quality coverage checks. |
Command Playbook#
node scripts/isis/run-v2-capability-benchmark.mjs
node scripts/isis/run-v2-ux-interaction-benchmark.mjs
node scripts/isis/run-v2-control-mode-quality-benchmark.mjs
node scripts/isis/generate-v2-benchmark-dashboard.mjs
node scripts/isis/validate-v2-benchmark-regression-gate.mjs
Source References#
scripts: scripts/isis/run-v2-capability-benchmark.mjs,
scripts/isis/run-v2-ux-interaction-benchmark.mjs,
scripts/isis/run-v2-control-mode-quality-benchmark.mjs,
scripts/isis/generate-v2-benchmark-dashboard.mjs,
scripts/isis/validate-v2-benchmark-regression-gate.mjs
benchmarkArtifacts:
docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json,
docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json,
docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json,
docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json,
docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json,
docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md