Scope#
WS14-008 implements synthetic transaction telemetry for critical RunPod
workflow families so operations can continuously detect regressions in
availability, latency, timeout behavior, and freshness.
Research and Existing Implementation Alignment#
Before implementation, existing operations/alerting paths were reviewed:
- RunPod telemetry snapshots and anomaly evaluators:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts
- Existing operations propagation path:
apps/isis/generation-api/src/services/comfyui.service.tsapps/isis/generation-api/src/routes/jobs.ts
- Existing SDK/date parsing and dashboard surfaces:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.tsapps/isis/web/src/pages/RunPodPage.tsx
No synthetic-transaction workflow-family snapshot existed, so this task adds a new first-class telemetry model integrated with existing RunPod operations.
Deliverables#
- Added synthetic transaction contracts in ai-providers:
libs/isis/ai-providers/src/providers/comfy/types.tslibs/isis/ai-providers/src/providers/comfy/index.ts
Added:
- status model (
pass,degraded,fail,missing), - objective model for critical families and SLO-like thresholds,
- per-family synthetic transaction entry model,
- synthetic transaction snapshot envelope.
- Implemented synthetic transaction evaluation in RunPod adapter:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts
Changes:
- Added synthetic transaction config controls:
syntheticTransactionsEnabled,syntheticTransactionCriticalWorkflowFamilies,syntheticTransactionWindowSize,syntheticTransactionMinimumSampleSize,syntheticTransactionSuccessRateMinPct,syntheticTransactionTimeoutRateMaxPct,syntheticTransactionLatencyP95Ms,syntheticTransactionFreshnessMaxAgeMinutes.
- Added
getSyntheticTransactionSnapshot()that evaluates each critical workflow family against success, timeout, latency, sample-size, and freshness checks. - Added deterministic status/reason classification for each family, with severity ordering and endpoint linkage when samples exist.
- Added adapter synthetic-transaction coverage:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.test.ts
New tests verify pass/fail/missing classification across critical workflow families.
- Exposed synthetic transactions through generation-api:
apps/isis/generation-api/src/services/comfyui.service.tsapps/isis/generation-api/src/routes/jobs.tsapps/isis/generation-api/src/routes/jobs.spec.ts
Changes:
- Added service accessor for synthetic transaction snapshot,
- returned
runpod.syntheticTransactionson RunPod operations endpoint, - added route contract fixture/assertions for synthetic transaction payloads.
- Extended SDK contracts/date parsing:
libs/isis/client/src/types.tslibs/isis/client/src/resources/generation.ts
Changes:
- added synthetic transaction interfaces,
- added date parsing for snapshot generated time and per-family last observation.
- Added RunPod dashboard synthetic transaction section:
apps/isis/web/src/pages/RunPodPage.tsxapps/isis/web/src/pages/RunPodPage.spec.tsx
Dashboard now surfaces:
- critical-family objectives and thresholds,
- per-family synthetic transaction status,
- sample size, success/timeout rates, p95 latency, last observation timestamp,
- failure/degradation reasons.
Verification Commands#
bash
pnpm --filter @isis/ai-providers exec vitest run src/providers/comfy/runpod-adapter.test.ts --reporter=dot
pnpm --filter @isis/generation-api exec vitest run src/routes/jobs.spec.ts --reporter=dot
pnpm --filter @isis/web exec vitest run src/pages/RunPodPage.spec.tsx --reporter=dot
pnpm --filter @isis/ai-providers run typecheck
pnpm --filter @isis/generation-api run typecheck
pnpm --filter @isis/client run typecheck
pnpm --filter @isis/web run typecheck
node scripts/isis/generate-remediation-backlog-v2.mjs
node scripts/isis/validate-v2-compliance-gate.mjs
Verification Notes#
@isis/ai-providersand@isis/generation-apitests pass with synthetic transaction coverage and route assertions.@isis/ai-providers,@isis/generation-api, and@isis/clienttypechecks pass.@isis/webRunPodPage.spec.tsxremains blocked by the pre-existing@isis/clientpackage-entry resolution issue.@isis/webtypecheck remains at known baseline failures outsideWS14-008scope (libs/isis/client/src/resources/workflows.tstyping and existingrootDirconstraints).