Eve library

ISIS v2 WS15-004: Golden-Output Test Sets with Perceptual Diff Thresholds

Guide

Revision 1

On this page

Scope#

WS15-004 introduces deterministic golden-output regression testing with perceptual diff thresholds for reproducibility-critical ISIS workflows.

The implementation adds:

  • reusable golden-output perceptual regression harness contracts,
  • deterministic baseline/candidate test sets for image and video-keyframe workflows,
  • threshold-gated pass/fail classification with case-level breach diagnostics,
  • CI enforcement of the perceptual regression contract suite.

Research and Existing Implementation Alignment#

Before implementing WS15-004, existing repository patterns were reviewed:

  • regression harness patterns:
    • libs/isis/audio-generation/src/qa/audio-regression-harness.ts
    • libs/isis/3d-generation/src/processing/three-d-benchmark-suite.ts
  • existing ISIS quality/consistency scoring:
    • libs/isis/workflows/src/consistency/temporal-quality-qa.ts
  • existing perceptual/pixel diff approaches:
    • libs/iris/testing/visual/src/pixel-diff-analyzer.ts

This alignment ensured the new harness follows existing ISIS deterministic benchmark semantics (expectedValid, expectation accuracy, threshold gating) while introducing explicit perceptual metrics for golden-output diffs.

Deliverables#

  1. Added perceptual golden-output regression harness in @isis/workflows
  • libs/isis/workflows/src/consistency/golden-output-perceptual-regression.ts

Implemented:

  • deterministic synthetic golden-output dataset builder,
  • perceptual metrics:
    • mean absolute error (normalized),
    • RMSE and PSNR,
    • global SSIM-style structural similarity,
    • edge/structure drift scoring,
    • aggregate perceptual score,
  • artifact-aware threshold profiles (image vs video-keyframe),
  • case-level breach diagnostics,
  • suite-level threshold gate evaluation (expectation accuracy, pass rate, average perceptual score).
  1. Added regression contract tests
  • libs/isis/workflows/src/consistency/golden-output-perceptual-regression.spec.ts

Coverage includes:

  • default deterministic suite pass behavior,
  • explicit threshold-gate regression failure behavior,
  • case-level perceptual metric diagnostics for triage/remediation.
  1. Added package-surface exports for harness consumption
  • libs/isis/workflows/src/consistency/index.ts
  • libs/isis/workflows/src/index.ts
  1. Added CI gate
  • .github/workflows/ci.yml

Added step in isis-api-route-contracts job:

  • pnpm --filter @isis/workflows exec vitest run src/consistency/golden-output-perceptual-regression.spec.ts

Verification Commands#

bash
pnpm --filter @isis/workflows exec vitest run src/consistency/golden-output-perceptual-regression.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Verification Notes#

  • The suite includes both expected-valid and expected-invalid golden-output comparisons, ensuring perceptual threshold contracts detect controlled regressions deterministically.
  • CI now enforces perceptual golden-output contracts as part of ISIS API contract gate execution.