Eve library

ISIS v2 WS08-010: Audio QA and Regression Harness

Guide

Revision 1

On this page

Scope#

WS08-010 introduces deterministic QA benchmark suites for:

  • Voice consistency stability across repeated synthesis runs.
  • Audio render reproducibility for fixed-seed rerenders.

The goal is to provide a repeatable regression harness with case-level outputs, threshold gates, and explicit failure surfaces for remediation workflows.

Research and Existing Implementation Alignment#

Before implementation, existing benchmark patterns were reviewed:

  • libs/isis/3d-generation/src/processing/three-d-benchmark-suite.ts
  • libs/isis/3d-generation/src/processing/three-d-benchmark-suite.spec.ts

These patterns were reused to keep:

  • deterministic dataset-driven benchmark execution,
  • expectation accuracy tracking (expectedValid vs observed validity),
  • threshold-based pass/fail summary semantics,
  • case-level failure ID reporting.

Deliverables#

  1. Added deterministic audio regression harness module
  • libs/isis/audio-generation/src/qa/audio-regression-harness.ts

Implemented:

  • voice consistency benchmark suite
  • render reproducibility benchmark suite
  • configurable per-suite scoring options and thresholds
  • aggregate report generation with failed threshold diagnostics
  • default benchmark dataset with expected pass/fail cases
  1. Added regression test coverage
  • libs/isis/audio-generation/src/qa/audio-regression-harness.spec.ts

Coverage includes:

  • default suite pass behavior
  • threshold regression failure behavior
  • case-level diagnostics availability
  1. Exported harness in package surfaces
  • libs/isis/audio-generation/src/qa/index.ts
  • libs/isis/audio-generation/src/index.ts
  • libs/isis/audio-generation/package.json (./qa subpath export)

Verification Commands#

bash
pnpm --filter @isis/audio-generation exec vitest run src/qa/audio-regression-harness.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Result#

WS08-010 now has a deterministic QA/regression harness for audio consistency and render reproducibility with explicit threshold gating and remediation-ready case diagnostics.