Scope#
WS15-003 adds deterministic replay tests for reproducibility-sensitive Isis
workflows, validating that fixed-seed executions replay to equivalent outputs
and seed changes produce deterministic divergence.
Research and Existing Implementation Alignment#
Before implementation, existing reproducibility and contract harnesses were reviewed:
- Existing workflow schema compatibility harness:
libs/isis/workflows/src/schema-compatibility-harness.spec.ts
- Existing workflow executor certification suite:
apps/isis/generation-api/src/workflows/executor.spec.ts
- Existing seed and reproducibility primitives:
libs/isis/workflows/src/seed/seed-manager.tslibs/isis/workflows/src/workflows.spec.ts
- Existing CI contract gates:
.github/workflows/ci.yml
This ensured replay tests were added as execution-level deterministic contracts instead of duplicating lower-level seed utility assertions.
Deliverables#
- Added deterministic replay contract test suite:
apps/isis/generation-api/src/workflows/deterministic-replay.spec.ts
Replay contract behavior:
- runs simulated execution with
DISABLE_COMFYUI_SERVICE=true, - fixes replay clock with fake timers/system time for deterministic execution path control,
- validates fixed-seed replay equality for reproducibility-sensitive workflows:
consistent-character,animatediff-controlnet,style-transfer,
- validates deterministic drift on seed change (
seed + 1) for each replay scenario.
- Added CI gate execution:
.github/workflows/ci.yml
Added step in isis-api-route-contracts job:
pnpm --filter @isis/generation-api exec vitest run src/workflows/deterministic-replay.spec.ts
Verification Commands#
bash
pnpm --filter @isis/generation-api exec vitest run src/workflows/deterministic-replay.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
Verification Notes#
- Replay tests assert deterministic output equivalence under fixed seed and deterministic divergence under seed variation for targeted reproducibility-sensitive workflows.
- CI now enforces deterministic replay contracts as part of Isis API contract validation gates.