Scope#
WS15-004 introduces deterministic golden-output regression testing with
perceptual diff thresholds for reproducibility-critical ISIS workflows.
The implementation adds:
- reusable golden-output perceptual regression harness contracts,
- deterministic baseline/candidate test sets for image and video-keyframe workflows,
- threshold-gated pass/fail classification with case-level breach diagnostics,
- CI enforcement of the perceptual regression contract suite.
Research and Existing Implementation Alignment#
Before implementing WS15-004, existing repository patterns were reviewed:
- regression harness patterns:
libs/isis/audio-generation/src/qa/audio-regression-harness.tslibs/isis/3d-generation/src/processing/three-d-benchmark-suite.ts
- existing ISIS quality/consistency scoring:
libs/isis/workflows/src/consistency/temporal-quality-qa.ts
- existing perceptual/pixel diff approaches:
libs/iris/testing/visual/src/pixel-diff-analyzer.ts
This alignment ensured the new harness follows existing ISIS deterministic
benchmark semantics (expectedValid, expectation accuracy, threshold gating)
while introducing explicit perceptual metrics for golden-output diffs.
Deliverables#
- Added perceptual golden-output regression harness in
@isis/workflows
libs/isis/workflows/src/consistency/golden-output-perceptual-regression.ts
Implemented:
- deterministic synthetic golden-output dataset builder,
- perceptual metrics:
- mean absolute error (normalized),
- RMSE and PSNR,
- global SSIM-style structural similarity,
- edge/structure drift scoring,
- aggregate perceptual score,
- artifact-aware threshold profiles (
imagevsvideo-keyframe), - case-level breach diagnostics,
- suite-level threshold gate evaluation (expectation accuracy, pass rate, average perceptual score).
- Added regression contract tests
libs/isis/workflows/src/consistency/golden-output-perceptual-regression.spec.ts
Coverage includes:
- default deterministic suite pass behavior,
- explicit threshold-gate regression failure behavior,
- case-level perceptual metric diagnostics for triage/remediation.
- Added package-surface exports for harness consumption
libs/isis/workflows/src/consistency/index.tslibs/isis/workflows/src/index.ts
- Added CI gate
.github/workflows/ci.yml
Added step in isis-api-route-contracts job:
pnpm --filter @isis/workflows exec vitest run src/consistency/golden-output-perceptual-regression.spec.ts
Verification Commands#
bash
pnpm --filter @isis/workflows exec vitest run src/consistency/golden-output-perceptual-regression.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
Verification Notes#
- The suite includes both expected-valid and expected-invalid golden-output comparisons, ensuring perceptual threshold contracts detect controlled regressions deterministically.
- CI now enforces perceptual golden-output contracts as part of ISIS API contract gate execution.