Eve library

ISIS v2 WS15-003: Deterministic Replay Tests for Reproducibility-Sensitive Workflows

Guide

Revision 1

On this page

Scope#

WS15-003 adds deterministic replay tests for reproducibility-sensitive Isis workflows, validating that fixed-seed executions replay to equivalent outputs and seed changes produce deterministic divergence.

Research and Existing Implementation Alignment#

Before implementation, existing reproducibility and contract harnesses were reviewed:

  • Existing workflow schema compatibility harness:
    • libs/isis/workflows/src/schema-compatibility-harness.spec.ts
  • Existing workflow executor certification suite:
    • apps/isis/generation-api/src/workflows/executor.spec.ts
  • Existing seed and reproducibility primitives:
    • libs/isis/workflows/src/seed/seed-manager.ts
    • libs/isis/workflows/src/workflows.spec.ts
  • Existing CI contract gates:
    • .github/workflows/ci.yml

This ensured replay tests were added as execution-level deterministic contracts instead of duplicating lower-level seed utility assertions.

Deliverables#

  1. Added deterministic replay contract test suite:
  • apps/isis/generation-api/src/workflows/deterministic-replay.spec.ts

Replay contract behavior:

  • runs simulated execution with DISABLE_COMFYUI_SERVICE=true,
  • fixes replay clock with fake timers/system time for deterministic execution path control,
  • validates fixed-seed replay equality for reproducibility-sensitive workflows:
    • consistent-character,
    • animatediff-controlnet,
    • style-transfer,
  • validates deterministic drift on seed change (seed + 1) for each replay scenario.
  1. Added CI gate execution:
  • .github/workflows/ci.yml

Added step in isis-api-route-contracts job:

  • pnpm --filter @isis/generation-api exec vitest run src/workflows/deterministic-replay.spec.ts

Verification Commands#

bash
pnpm --filter @isis/generation-api exec vitest run src/workflows/deterministic-replay.spec.ts --reporter=dot
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Verification Notes#

  • Replay tests assert deterministic output equivalence under fixed seed and deterministic divergence under seed variation for targeted reproducibility-sensitive workflows.
  • CI now enforces deterministic replay contracts as part of Isis API contract validation gates.