# ISIS v2 WS14-008: Synthetic Transactions for Critical Workflow Families

## Scope

`WS14-008` implements synthetic transaction telemetry for critical RunPod
workflow families so operations can continuously detect regressions in
availability, latency, timeout behavior, and freshness.

## Research and Existing Implementation Alignment

Before implementation, existing operations/alerting paths were reviewed:

- RunPod telemetry snapshots and anomaly evaluators:
  - `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts`
- Existing operations propagation path:
  - `apps/isis/generation-api/src/services/comfyui.service.ts`
  - `apps/isis/generation-api/src/routes/jobs.ts`
- Existing SDK/date parsing and dashboard surfaces:
  - `libs/isis/client/src/types.ts`
  - `libs/isis/client/src/resources/generation.ts`
  - `apps/isis/web/src/pages/RunPodPage.tsx`

No synthetic-transaction workflow-family snapshot existed, so this task adds a
new first-class telemetry model integrated with existing RunPod operations.

## Deliverables

1. Added synthetic transaction contracts in ai-providers:

- `libs/isis/ai-providers/src/providers/comfy/types.ts`
- `libs/isis/ai-providers/src/providers/comfy/index.ts`

Added:

- status model (`pass`, `degraded`, `fail`, `missing`),
- objective model for critical families and SLO-like thresholds,
- per-family synthetic transaction entry model,
- synthetic transaction snapshot envelope.

2. Implemented synthetic transaction evaluation in RunPod adapter:

- `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts`

Changes:

- Added synthetic transaction config controls:
  - `syntheticTransactionsEnabled`,
  - `syntheticTransactionCriticalWorkflowFamilies`,
  - `syntheticTransactionWindowSize`,
  - `syntheticTransactionMinimumSampleSize`,
  - `syntheticTransactionSuccessRateMinPct`,
  - `syntheticTransactionTimeoutRateMaxPct`,
  - `syntheticTransactionLatencyP95Ms`,
  - `syntheticTransactionFreshnessMaxAgeMinutes`.
- Added `getSyntheticTransactionSnapshot()` that evaluates each critical
  workflow family against success, timeout, latency, sample-size, and freshness
  checks.
- Added deterministic status/reason classification for each family, with
  severity ordering and endpoint linkage when samples exist.

3. Added adapter synthetic-transaction coverage:

- `libs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts`

New tests verify pass/fail/missing classification across critical workflow
families.

4. Exposed synthetic transactions through generation-api:

- `apps/isis/generation-api/src/services/comfyui.service.ts`
- `apps/isis/generation-api/src/routes/jobs.ts`
- `apps/isis/generation-api/src/routes/jobs.spec.ts`

Changes:

- Added service accessor for synthetic transaction snapshot,
- returned `runpod.syntheticTransactions` on RunPod operations endpoint,
- added route contract fixture/assertions for synthetic transaction payloads.

5. Extended SDK contracts/date parsing:

- `libs/isis/client/src/types.ts`
- `libs/isis/client/src/resources/generation.ts`

Changes:

- added synthetic transaction interfaces,
- added date parsing for snapshot generated time and per-family last
  observation.

6. Added RunPod dashboard synthetic transaction section:

- `apps/isis/web/src/pages/RunPodPage.tsx`
- `apps/isis/web/src/pages/RunPodPage.spec.tsx`

Dashboard now surfaces:

- critical-family objectives and thresholds,
- per-family synthetic transaction status,
- sample size, success/timeout rates, p95 latency, last observation timestamp,
- failure/degradation reasons.

## Verification Commands

```bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
pnpm --filter @isis/ai-providers run typecheck
pnpm --filter @isis/generation-api run typecheck
pnpm --filter @isis/client run typecheck
pnpm --filter @isis/web run typecheck
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
```

## Verification Notes

- `@isis/ai-providers` and `@isis/generation-api` tests pass with synthetic
  transaction coverage and route assertions.
- `@isis/ai-providers`, `@isis/generation-api`, and `@isis/client` typechecks
  pass.
- `@isis/web` `RunPodPage.spec.tsx` remains blocked by the pre-existing
  `@isis/client` package-entry resolution issue.
- `@isis/web` typecheck remains at known baseline failures outside `WS14-008`
  scope (`libs/isis/client/src/resources/workflows.ts` typing and existing
  `rootDir` constraints).
