# ISIS v2 WS15-004: Golden-Output Test Sets with Perceptual Diff Thresholds

## Scope

`WS15-004` introduces deterministic golden-output regression testing with
perceptual diff thresholds for reproducibility-critical ISIS workflows.

The implementation adds:

- reusable golden-output perceptual regression harness contracts,
- deterministic baseline/candidate test sets for image and video-keyframe
  workflows,
- threshold-gated pass/fail classification with case-level breach diagnostics,
- CI enforcement of the perceptual regression contract suite.

## Research and Existing Implementation Alignment

Before implementing `WS15-004`, existing repository patterns were reviewed:

- regression harness patterns:
  - `libs/isis/audio-generation/src/qa/audio-regression-harness.ts`
  - `libs/isis/3d-generation/src/processing/three-d-benchmark-suite.ts`
- existing ISIS quality/consistency scoring:
  - `libs/isis/workflows/src/consistency/temporal-quality-qa.ts`
- existing perceptual/pixel diff approaches:
  - `libs/iris/testing/visual/src/pixel-diff-analyzer.ts`

This alignment ensured the new harness follows existing ISIS deterministic
benchmark semantics (`expectedValid`, expectation accuracy, threshold gating)
while introducing explicit perceptual metrics for golden-output diffs.

## Deliverables

1. Added perceptual golden-output regression harness in `@isis/workflows`

- `libs/isis/workflows/src/consistency/golden-output-perceptual-regression.ts`

Implemented:

- deterministic synthetic golden-output dataset builder,
- perceptual metrics:
  - mean absolute error (normalized),
  - RMSE and PSNR,
  - global SSIM-style structural similarity,
  - edge/structure drift scoring,
  - aggregate perceptual score,
- artifact-aware threshold profiles (`image` vs `video-keyframe`),
- case-level breach diagnostics,
- suite-level threshold gate evaluation (expectation accuracy, pass rate,
  average perceptual score).

2. Added regression contract tests

- `libs/isis/workflows/src/consistency/golden-output-perceptual-regression.spec.ts`

Coverage includes:

- default deterministic suite pass behavior,
- explicit threshold-gate regression failure behavior,
- case-level perceptual metric diagnostics for triage/remediation.

3. Added package-surface exports for harness consumption

- `libs/isis/workflows/src/consistency/index.ts`
- `libs/isis/workflows/src/index.ts`

4. Added CI gate

- `.github/workflows/ci.yml`

Added step in `isis-api-route-contracts` job:

- `pnpm --filter @isis/workflows exec vitest run src/consistency/golden-output-perceptual-regression.spec.ts`

## Verification Commands

```bash
pnpm --filter @isis/workflows exec vitest run src/consistency/golden-output-perceptual-regression.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

## Verification Notes

- The suite includes both expected-valid and expected-invalid golden-output
  comparisons, ensuring perceptual threshold contracts detect controlled
  regressions deterministically.
- CI now enforces perceptual golden-output contracts as part of ISIS API
  contract gate execution.
