Eve library

ISIS v2 Quality Benchmark Methodology and Interpretation Guide

Guide

Revision 1

On this page

Generated: 2026-03-06T11:20:13.891Z Scope: WS16-008

Summary#

  • Benchmark pipeline steps: 5
  • Regression assertions: 14
  • Failing assertions: 0
  • Control failure classes: 5
  • Methodology formulas captured: 6

Benchmark Pipeline Methodology#

Step Checklist Script Inputs Outputs
Capability benchmark WS01-007 scripts/isis/run-v2-capability-benchmark.mjs docs/domains/isis/remediation-v2/ISIS_V2_COMFY_CAPABILITY_MATRIX.json, libs/isis/workflows/src/workflows docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.md
UX interaction benchmark WS01-008 scripts/isis/run-v2-ux-interaction-benchmark.mjs apps/isis/web/src/pages/WorkflowEditorPage.tsx, apps/isis/web/src/pages/WorkflowTemplatesPage.tsx, apps/isis/web/src/pages/WorkflowsPage.tsx, docs/domains/isis/remediation-v2/ISIS_V2_ASSET_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_OUTPUT_PACKAGE_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json, libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.md
Control-mode quality benchmark WS05-009 scripts/isis/run-v2-control-mode-quality-benchmark.mjs docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json, libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.md
Benchmark dashboard aggregation WS01-009 scripts/isis/generate-v2-benchmark-dashboard.mjs docs/domains/isis/remediation-v2/ISIS_V2_ASSET_TAXONOMY.json, docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_MATRIX.json, libs/isis/workflows/src/metadata/workflow-taxonomy-registry.json docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.md
Benchmark regression gate WS01-010 scripts/isis/validate-v2-benchmark-regression-gate.mjs docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json, docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md

Metric Formulas#

Capability Benchmark#

  • nodeTargetPassRate = (requiredNodeTargets - failedRequiredNodeTargets) / requiredNodeTargets
  • workflowCapabilityPassRate = (requiredWorkflowCapabilities - missingRequiredCapabilities) / requiredWorkflowCapabilities

UX Benchmark#

  • requiredCapabilityPassRate = passedRequiredCapabilities / requiredCapabilities
  • workflowPassRate = workflowsPassed / workflowCoverage
  • taxonomyCategoryCoverage = matchedTaxonomyCategories / taxonomyCategories

Control-Mode Quality Scoring#

  • Pass threshold: 0.75
  • Weights: workflowPassRate=0.55, capabilityPassRate=0.3, modalityCoverage=0.15
  • Formula: qualityScore = workflowPassRateworkflowWeight + capabilityPassRatecapabilityWeight + modalityCoverage*modalityWeight

Regression Assertion Interpretation#

  • Baseline description: Minimum acceptable benchmark levels. Pass-rate metrics must not decrease; missing-gap metrics must not increase.
Assertion Metric Actual Rule Threshold Margin Result Interpretation
Capability workflow pass rate non-regression capability.workflowCapabilityPassRate 1 >= 0.7777777777777778 0.222222 PASS Within guardrail threshold.
Capability node-class pass rate non-regression capability.nodeTargetPassRate 0.9166666666666666 >= 0.8333333333333334 0.083333 PASS Within guardrail threshold.
Missing required workflow capabilities must not increase capability.missingRequiredCapabilitiesCount 0 <= 4 4.000000 PASS Within guardrail threshold.
Failed required node targets must not increase capability.failedRequiredNodeTargetsCount 1 <= 2 1.000000 PASS Within guardrail threshold.
UX required capability pass rate non-regression ux.requiredCapabilityPassRate 1 >= 1 0.000000 PASS At threshold. Treat as fragile and monitor closely.
UX workflow pass rate non-regression ux.workflowPassRate 1 >= 1 0.000000 PASS At threshold. Treat as fragile and monitor closely.
UX passed workflow count non-regression ux.workflowsPassed 48 >= 48 0.000000 PASS At threshold. Treat as fragile and monitor closely.
Missing required UX capabilities must not increase ux.missingRequiredCapabilityCount 0 <= 0 0.000000 PASS At threshold. Treat as fragile and monitor closely.
Missing UX capability bucket count must not increase ux.missingCapabilityBuckets 0 <= 0 0.000000 PASS At threshold. Treat as fragile and monitor closely.
Taxonomy category coverage non-regression ux.taxonomyCategoryCoverage 1 >= 1 0.000000 PASS At threshold. Treat as fragile and monitor closely.
Control-mode average quality score non-regression control.averageQualityScore 0.972039 >= 0.251944 0.720095 PASS Within guardrail threshold.
Failing control-mode count must not increase control.failingControlCount 7 <= 27 20.000000 PASS Within guardrail threshold.
High-severity control failures must not increase control.highSeverityFailureCount 0 <= 27 27.000000 PASS Within guardrail threshold.
Control modes with no workflow coverage must not increase control.noWorkflowCoverageCount 0 <= 5 5.000000 PASS Within guardrail threshold.

Control Failure Taxonomy#

Class Severity Description
benchmark-score-gap medium Composite benchmark score is below pass threshold (0.75).
modality-coverage-gap medium Observed workflow modality coverage for the control does not match declared supported modalities.
no-workflow-coverage critical Control modality is defined in taxonomy but not used by any workflow.
ux-capability-gap high Control-specific UX capability mappings are missing or failing benchmark probes.
workflow-quality-gap high One or more workflows using this control fail required UX quality coverage checks.

Command Playbook#

  • node scripts/isis/run-v2-capability-benchmark.mjs
  • node scripts/isis/run-v2-ux-interaction-benchmark.mjs
  • node scripts/isis/run-v2-control-mode-quality-benchmark.mjs
  • node scripts/isis/generate-v2-benchmark-dashboard.mjs
  • node scripts/isis/validate-v2-benchmark-regression-gate.mjs

Source References#

  • scripts: scripts/isis/run-v2-capability-benchmark.mjs, scripts/isis/run-v2-ux-interaction-benchmark.mjs, scripts/isis/run-v2-control-mode-quality-benchmark.mjs, scripts/isis/generate-v2-benchmark-dashboard.mjs, scripts/isis/validate-v2-benchmark-regression-gate.mjs
  • benchmarkArtifacts: docs/domains/isis/remediation-v2/ISIS_V2_CAPABILITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_CONTROL_MODE_QUALITY_BENCHMARK.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_BASELINE.json, docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_REGRESSION_ASSERTIONS.md