Eve library

ISIS v2 WS14-008: Synthetic Transactions for Critical Workflow Families

Guide

Revision 1

On this page

Scope#

WS14-008 implements synthetic transaction telemetry for critical RunPod workflow families so operations can continuously detect regressions in availability, latency, timeout behavior, and freshness.

Research and Existing Implementation Alignment#

Before implementation, existing operations/alerting paths were reviewed:

  • RunPod telemetry snapshots and anomaly evaluators:
    • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts
  • Existing operations propagation path:
    • apps/isis/generation-api/src/services/comfyui.service.ts
    • apps/isis/generation-api/src/routes/jobs.ts
  • Existing SDK/date parsing and dashboard surfaces:
    • libs/isis/client/src/types.ts
    • libs/isis/client/src/resources/generation.ts
    • apps/isis/web/src/pages/RunPodPage.tsx

No synthetic-transaction workflow-family snapshot existed, so this task adds a new first-class telemetry model integrated with existing RunPod operations.

Deliverables#

  1. Added synthetic transaction contracts in ai-providers:
  • libs/isis/ai-providers/src/providers/comfy/types.ts
  • libs/isis/ai-providers/src/providers/comfy/index.ts

Added:

  • status model (pass, degraded, fail, missing),
  • objective model for critical families and SLO-like thresholds,
  • per-family synthetic transaction entry model,
  • synthetic transaction snapshot envelope.
  1. Implemented synthetic transaction evaluation in RunPod adapter:
  • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts

Changes:

  • Added synthetic transaction config controls:
    • syntheticTransactionsEnabled,
    • syntheticTransactionCriticalWorkflowFamilies,
    • syntheticTransactionWindowSize,
    • syntheticTransactionMinimumSampleSize,
    • syntheticTransactionSuccessRateMinPct,
    • syntheticTransactionTimeoutRateMaxPct,
    • syntheticTransactionLatencyP95Ms,
    • syntheticTransactionFreshnessMaxAgeMinutes.
  • Added getSyntheticTransactionSnapshot() that evaluates each critical workflow family against success, timeout, latency, sample-size, and freshness checks.
  • Added deterministic status/reason classification for each family, with severity ordering and endpoint linkage when samples exist.
  1. Added adapter synthetic-transaction coverage:
  • libs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts

New tests verify pass/fail/missing classification across critical workflow families.

  1. Exposed synthetic transactions through generation-api:
  • apps/isis/generation-api/src/services/comfyui.service.ts
  • apps/isis/generation-api/src/routes/jobs.ts
  • apps/isis/generation-api/src/routes/jobs.spec.ts

Changes:

  • Added service accessor for synthetic transaction snapshot,
  • returned runpod.syntheticTransactions on RunPod operations endpoint,
  • added route contract fixture/assertions for synthetic transaction payloads.
  1. Extended SDK contracts/date parsing:
  • libs/isis/client/src/types.ts
  • libs/isis/client/src/resources/generation.ts

Changes:

  • added synthetic transaction interfaces,
  • added date parsing for snapshot generated time and per-family last observation.
  1. Added RunPod dashboard synthetic transaction section:
  • apps/isis/web/src/pages/RunPodPage.tsx
  • apps/isis/web/src/pages/RunPodPage.spec.tsx

Dashboard now surfaces:

  • critical-family objectives and thresholds,
  • per-family synthetic transaction status,
  • sample size, success/timeout rates, p95 latency, last observation timestamp,
  • failure/degradation reasons.

Verification Commands#

bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
pnpm --filter @isis/ai-providers run typecheck
pnpm --filter @isis/generation-api run typecheck
pnpm --filter @isis/client run typecheck
pnpm --filter @isis/web run typecheck
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs

Verification Notes#

  • @isis/ai-providers and @isis/generation-api tests pass with synthetic transaction coverage and route assertions.
  • @isis/ai-providers, @isis/generation-api, and @isis/client typechecks pass.
  • @isis/web RunPodPage.spec.tsx remains blocked by the pre-existing @isis/client package-entry resolution issue.
  • @isis/web typecheck remains at known baseline failures outside WS14-008 scope (libs/isis/client/src/resources/workflows.ts typing and existing rootDir constraints).