# ISIS_TODOS.md

Last Updated: 2026-02-21 Source Audit:
`docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md`

## Objective

Close every Isis domain gap identified in the audit and deliver production-grade
end-to-end workflow execution, RunPod pay-per-second serverless operations, and
complete Isis web app feature parity.

## Global Definition of Done

- [x] All P0, P1, and P2 findings from the audit are closed with linked PRs.
- [x] No simulated/placeholder runtime behavior remains in production profiles.
- [x] API, SDK, worker, and web app contracts are aligned and versioned.
- [x] Full observability exists for job lifecycle, cost, latency, and failure
      modes.
- [x] CI gates block regressions on schema compatibility, execution path, and
      provider integrations.
- [x] All 45 workflow IDs are certified end-to-end (schema -> execution ->
      RunPod -> web app -> E2E).

## Workstream 00: Program Governance and Delivery Controls

- [x] [WS00-001] Create an Isis remediation epic with one ticket per checklist
      item.
- [x] [WS00-002] Assign accountable owner for each finding (F01-F22).
- [x] [WS00-003] Define target milestones for Phase 0/1/2/3 with explicit dates.
- [x] [WS00-004] Create a risk board for execution, infra, and cost regressions.
- [x] [WS00-005] Add architecture decision records (ADRs) for queueing model,
      schema canonicalization, and RunPod strategy.
- [x] [WS00-006] Define rollback criteria for all production rollouts.
- [x] [WS00-007] Define release quality bar and sign-off matrix (backend,
      platform, frontend, SRE, security).
- [x] [WS00-008] Add weekly status report template with burn-down by finding.

## Workstream 01 (F01): Job Submission to Execution Pipeline (P0)

- [x] [F01-001] Define canonical job envelope schema in a shared package
      (`jobId`, `type`, `input`, `priority`, `metadata`, `idempotencyKey`).
- [x] [F01-002] Add BullMQ queue producer module to generation API.
- [x] [F01-003] Publish queue name and redis config as strict required env vars
      for production.
- [x] [F01-004] Enqueue on submit and persist status transition to `queued`.
- [x] [F01-005] Add transactional outbox pattern so DB write and enqueue cannot
      diverge.
- [x] [F01-006] Add idempotency key handling to prevent duplicate enqueues.
- [x] [F01-007] Ensure cancellation removes queued jobs and marks cancelled
      reason.
- [x] [F01-008] Add retry policy with capped attempts and exponential backoff.
- [x] [F01-009] Add dead-letter queue with retention policy and replay tooling.
- [x] [F01-010] Emit lifecycle events for
      queued/running/progress/completed/failed/cancelled from real execution
      path.
- [x] [F01-011] Ensure worker ACK/NACK semantics update API-side job status
      deterministically.
- [x] [F01-012] Add queue position endpoint backed by queue state, not only DB
      estimation.
- [x] [F01-013] Add queue drain/health admin operation with auth guard.
- [x] [F01-014] Add migration/cleanup for legacy `pending` jobs stuck before
      queue wiring.
- [x] [F01-015] Add load tests for enqueue throughput and worker consumption.

## Workstream 02 (F02): Workflow Catalog Runtime Integration (P0)

- [x] [F02-001] Add startup bootstrap to register all entries from
      `@isis/workflows` into executor resolver.
- [x] [F02-002] Remove hardcoded executor default IDs not present in registry.
- [x] [F02-003] Map each generation type to valid catalog workflow IDs.
- [x] [F02-004] Add validation to fail startup if mapped defaults are missing.
- [x] [F02-005] Add runtime guard to fail closed on unresolved workflow in
      production.
- [x] [F02-006] Add per-workflow capability metadata (input schema, output
      schema, provider compatibility).
- [x] [F02-007] Add deterministic workflow selection policy (user override >
      project default > system default).
- [x] [F02-008] Add workflow cache warm-up at service boot.
- [x] [F02-009] Add resolver metrics (`resolve_success`, `resolve_failure`,
      `fallback_used`).
- [x] [F02-010] Remove placeholder output URL generation in executor production
      path.

## Workstream 03 (F03): Canonical Workflow Schema and Interop (P0)

- [x] [F03-001] Choose canonical Isis workflow format (object-map nodes vs array
      nodes/links).
- [x] [F03-002] Publish schema v1.0 with strict compatibility rules.
- [x] [F03-003] Refactor workflow-registry validator to support canonical
      schema.
- [x] [F03-004] If dual support is required, implement reversible adapter with
      lossless conversion tests.
- [x] [F03-005] Migrate all 45 workflows to canonical format if needed.
- [x] [F03-006] Add contract test: every workflow in
      `libs/isis/workflows/src/workflows` passes registry validation.
- [x] [F03-007] Add contract test: registry-exported workflow can execute via
      executor without schema mutation errors.
- [x] [F03-008] Add schema lint rule in CI to block non-canonical workflow
      files.
- [x] [F03-009] Add schema version field and migration registry.
- [x] [F03-010] Add docs for custom node contracts and required fields.

## Workstream 04 (F04): Isis Web App Full Feature Exposure (P0)

- [x] [F04-001] Create `apps/isis/web` application scaffold with routing, auth,
      and API client wiring.
- [x] [F04-002] Add tenant-aware authentication and RBAC.
- [x] [F04-003] Add jobs dashboard (list, filter, sort, status, cancel, retry,
      delete).
- [x] [F04-004] Add job submission studio supporting all generation types.
- [x] [F04-005] Add workflow catalog page (search/filter/category/tag/GPU
      requirements).
- [x] [F04-006] Add workflow details page (metadata, schema, node graph
      preview).
- [x] [F04-007] Add workflow create/edit UI with validation feedback.
- [x] [F04-008] Add workflow version management UI
      (create/list/get/deprecate/activate).
- [x] [F04-009] Add templates page and create-from-template flow.
- [x] [F04-010] Add workflow run UI with parameter forms generated from schema.
- [x] [F04-011] Add outputs registry page (list/get/update/delete/batch-delete).
- [x] [F04-012] Add output storage operations UI (download URL, upload URL, move
      tier, restore).
- [x] [F04-013] Add provenance editor and visualization.
- [x] [F04-014] Add lineage graph explorer (ancestors/descendants/both, depth
      controls).
- [x] [F04-015] Add retention policy management UI.
- [x] [F04-016] Add models page (register, upload complete, list, update,
      delete, download).
- [x] [F04-017] Add provider operations page (queue stats, health, failover
      status).
- [x] [F04-018] Add RunPod operations page (cost, usage, active endpoints,
      failures).
- [x] [F04-019] Add live job progress updates via polling or SSE.
- [x] [F04-020] Add complete audit log panel for workflow and model changes.
- [x] [F04-021] Add feature flags panel (read-only in prod) for provider routing
      diagnostics.
- [x] [F04-022] Add error state UX for all failure categories
      (validation/network/provider/auth).
- [x] [F04-023] Add empty states, skeleton loading states, and retry actions
      across all views.
- [x] [F04-024] Add mobile-responsive layouts for core operational pages.
- [x] [F04-025] Add accessibility compliance checks (keyboard navigation, aria
      labels, contrast).

## Workstream 05 (F05): RunPod Adapter Productionization (P1)

- [x] [F05-001] Implement real `estimateCost()` using GPU pricing + measured
      runtime estimates.
- [x] [F05-002] Implement real `getCredits()` integration path (or internal
      wallet abstraction with source-of-truth sync).
- [x] [F05-003] Implement persistent `getJobHistory()` with indexed query
      support.
- [x] [F05-004] Implement model inventory endpoint for baked endpoint images.
- [x] [F05-005] Implement custom node inventory endpoint for baked endpoint
      images.
- [x] [F05-006] Replace synthetic timestamps with provider-sourced timestamps
      where available.
- [x] [F05-007] Add status mapping contract tests for every RunPod status
      variant.
- [x] [F05-008] Add retry/circuit-breaker tests for rate limit, timeout, auth
      failure, and server error.
- [x] [F05-009] Add provider error taxonomy mapping to user-safe messages and
      machine-readable codes.
- [x] [F05-010] Add tracing context propagation (`correlationId`, `projectId`,
      `userId`).

## Workstream 06 (F06): RunPod Handler Capability Expansion (P1)

- [x] [F06-001] Add workflow-id driven execution mode for catalog-native
      invocation.
- [x] [F06-002] Add endpoint that returns supported workflow IDs and metadata.
- [x] [F06-003] Add typed contracts for major Isis categories (video, 3d,
      fractal, avatar, sacred-art, utility).
- [x] [F06-004] Add schema validation for workflow overrides and input images
      per workflow.
- [x] [F06-005] Add compatibility checks for required custom nodes/models before
      execution.
- [x] [F06-006] Add deterministic error response format for unsupported
      workflow/category.
- [x] [F06-007] Add execution limits per workflow class
      (timeout/memory/frame/steps).
- [x] [F06-008] Add test fixtures for each major category.

## Workstream 07 (F07): Remove Simulated Runtime Behavior from Executors (P1)

- [x] [F07-001] Add production guard: simulated paths must be disabled in
      production builds.
- [x] [F07-002] Texture Upscale: integrate real model inference backend and
      output validation.
- [x] [F07-003] Gaussian Splatting: integrate real reconstruction pipeline and
      real output serializers.
- [x] [F07-004] Blender Render: enforce Blender dependency check at startup and
      fail fast if missing.
- [x] [F07-005] Mesh Processing: replace simulated operations with real
      toolchain operations.
- [x] [F07-006] Add golden-output tests for each executor.
- [x] [F07-007] Add output checksum and file integrity verification after
      execution.
- [x] [F07-008] Add performance benchmarks and min throughput targets per
      executor.
- [x] [F07-009] Add cancellation tests for long-running operations.
- [x] [F07-010] Add resource leak tests (VRAM, temp files, subprocess cleanup).

## Workstream 08 (F08): RunPod Provisioning and Drift Control (P1)

- [x] [F08-001] Decide provisioning mechanism (Terraform provider/external
      module/API-driven CI job).
- [x] [F08-002] Implement endpoint lifecycle automation (create/update/delete).
- [x] [F08-003] Store endpoint desired state in code (GPU type, min/max workers,
      idle timeout, execution timeout).
- [x] [F08-004] Add drift detection comparing desired state vs actual endpoint
      config.
- [x] [F08-005] Add reconcile command for drift remediation.
- [x] [F08-006] Add CI validation for runpod endpoint definitions before deploy.
- [x] [F08-007] Add staged rollout by endpoint with canary traffic percentages.
- [x] [F08-008] Add on-call runbook for endpoint outages and cost spikes.

## Workstream 09 (F09): Workflow Registry Persistence and Auth Hardening (P2)

- [x] [F09-001] Replace in-memory storage with PostgreSQL persistence.
- [x] [F09-002] Add migrations for workflows, versions, templates, stars, runs,
      and ownership tables.
- [x] [F09-003] Remove `DEFAULT_OWNER = 'user_demo'` and source owner from auth
      claims.
- [x] [F09-004] Add RBAC checks for create/update/delete/version operations.
- [x] [F09-005] Add optimistic locking/version conflict handling on updates.
- [x] [F09-006] Add pagination + indexing for large catalogs.
- [x] [F09-007] Add soft-delete + restore support for workflows and versions.
- [x] [F09-008] Add audit trail for all write operations.

## Workstream 10 (F10): Output Registry Ownership and Access Control (P2)

- [x] [F10-001] Remove `DEFAULT_OWNER = 'user_demo'` and enforce authenticated
      owner context.
- [x] [F10-002] Add row-level authorization checks for
      outputs/provenance/lineage.
- [x] [F10-003] Add tenant scoping in all list and graph queries.
- [x] [F10-004] Add policy enforcement for cross-project lineage links.
- [x] [F10-005] Add audit logging for provenance and lineage mutations.

## Workstream 11 (F11): Health and Readiness Truthfulness (P2)

- [x] [F11-001] Replace placeholder readiness values for queue/storage with real
      checks.
- [x] [F11-002] Add Redis connectivity and queue liveness probe.
- [x] [F11-003] Add object storage read/write probe with timeout and retries.
- [x] [F11-004] Add provider health check with dependency-specific status
      details.
- [x] [F11-005] Ensure readiness endpoint returns `503` on critical dependency
      failure.
- [x] [F11-006] Add alerting based on readiness degradation.

## Workstream 12 (F12): Model Storage Presign Productionization (P2)

- [x] [F12-001] Integrate real pre-signed upload URL generation via
      `@oshun/storage`.
- [x] [F12-002] Integrate real pre-signed download URL generation via
      `@oshun/storage`.
- [x] [F12-003] Add content hash verification step on `completeUpload`.
- [x] [F12-004] Enforce MIME/type and size constraints at upload and completion.
- [x] [F12-005] Add anti-virus/malware scan hook for uploaded model files.
- [x] [F12-006] Add retention/cleanup policy for abandoned uploads.

## Workstream 13 (F13): Deployment Configuration Drift (P2)

- [x] [F13-001] Update docker compose Isis app paths to
      `apps/isis/generation-api` and `apps/isis/gpu-worker`.
- [x] [F13-002] Add CI check to validate `APP_PATH` values exist for all compose
      services.
- [x] [F13-003] Add smoke test that boots Isis services via compose and verifies
      health endpoints.
- [x] [F13-004] Document local startup workflow for Isis services and GPU
      profile.

## Workstream 14 (F14): Test Coverage Expansion for Real Execution (P2)

- [x] [F14-001] Add full E2E test: submit job -> queue -> worker -> output ->
      completed state.
- [x] [F14-002] Add E2E test for cancel during queued and running states.
- [x] [F14-003] Add E2E test for retry flow with bounded attempts.
- [x] [F14-004] Add contract test for all 45 workflows executing validation +
      resolution.
- [x] [F14-005] Add RunPod sandbox integration tests for submit/status/cancel.
- [x] [F14-006] Add cost-accounting correctness tests with fixture runtimes.
- [x] [F14-007] Add web app E2E tests covering every major interaction surface.
- [x] [F14-008] Add chaos tests for provider failover and queue outage handling.

## Workstream 15 (F15): API and SDK Contract Parity (Critical Completeness)

- [x] [F15-001] Inventory all `@isis/client` methods and map to live API routes.
- [x] [F15-002] Resolve generation route mismatches (`/api/v1/jobs/batch`,
      `/api/v1/jobs/:id/retry`, `/api/v1/queue/stats`,
      `/api/v1/jobs/:id/queue-position`).
- [x] [F15-003] Resolve workflows route mismatches
      (metadata/clone/activate/tag/version validation endpoints).
- [x] [F15-004] Resolve outputs route mismatches (batch-delete,
      download/upload-url, move-tier, restore, provenance update, policy
      endpoints).
- [x] [F15-005] Decide whether to add missing endpoints or deprecate unsupported
      SDK methods.
- [x] [F15-006] Add OpenAPI specs for generation/workflow/output services.
- [x] [F15-007] Generate SDK from OpenAPI or add CI contract tests to prevent
      drift.
- [x] [F15-008] Add breaking-change detection job in CI for API route contracts.

## Workstream 16: Security and Multi-Tenant Hardening

- [x] [SEC-001] Add auth middleware consistency across generation, workflow,
      output, and model endpoints.
- [x] [SEC-002] Enforce project-level authorization on every read/write route.
- [x] [SEC-003] Add API key scopes for admin/ops/provider-control actions.
- [x] [SEC-004] Add webhook signature verification for provider callbacks.
- [x] [SEC-005] Add input sanitization and payload size limits on workflow
      definitions.
- [x] [SEC-006] Add structured security logs for sensitive operations.
- [x] [SEC-007] Add abuse controls for expensive generation endpoints.

## Workstream 17: Observability, Cost, and SLOs

- [x] [OBS-001] Define SLOs for submit latency, queue delay, execution success
      rate, and end-to-end completion time.
- [x] [OBS-002] Add metrics for queue depth, stuck jobs, execution retries, and
      DLQ size.
- [x] [OBS-003] Add per-provider latency and failure dashboards.
- [x] [OBS-004] Add cost dashboards by endpoint, project, workflow, and user.
- [x] [OBS-005] Add anomaly alerts for cost spikes and failure rates.
- [x] [OBS-006] Add distributed tracing across API -> queue -> worker ->
      provider -> storage.
- [x] [OBS-007] Add operator CLI commands for stuck-job triage and replay.

## Workstream 18: Data Migration and Backfill

- [x] [MIG-001] Design migration for existing jobs to new lifecycle/status
      model.
- [x] [MIG-002] Backfill workflow IDs for historical jobs where missing.
- [x] [MIG-003] Backfill provenance records for completed outputs where
      possible.
- [x] [MIG-004] Backfill cost data from historical provider logs.
- [x] [MIG-005] Add migration rollback and verification scripts.

## Workstream 19: Release Plan and Cutover

- [x] [REL-001] Enable queue producer behind feature flag for safe rollout.
- [x] [REL-002] Canary release to limited traffic/project set.
- [x] [REL-003] Verify SLO and cost behavior during canary.
- [x] [REL-004] Gradually increase rollout percentage with auto-rollback
      thresholds.
- [x] [REL-005] Conduct game-day exercises before full cutover.
- [x] [REL-006] Finalize post-cutover validation checklist and sign-off.

## Workstream 20: Per-Workflow End-to-End Certification (45/45)

- [x] [WF-001] 3d-texture: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-002] animatediff-controlnet: certify schema validity, resolver
      mapping, executor success, RunPod success, web app run path, and E2E
      regression.
- [x] [WF-003] artifact-reconstruction: certify schema validity, resolver
      mapping, executor success, RunPod success, web app run path, and E2E
      regression.
- [x] [WF-004] background-removal: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-005] barnsley-fern: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-006] base-avatar: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-007] batch-resize: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-008] batch-watermark: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-009] buddhist-thangka: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-010] celtic-knotwork: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-011] christian-icon: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-012] concept-explainer: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-013] consciousness-map: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-014] consistent-character: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-015] cosmetic-layer: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-016] egyptian-hieratic: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-017] environment-map: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-018] face-swap: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-019] flame-fractal: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-020] fractal-art-fusion: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-021] fractal-morph: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-022] fractal-zoom: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-023] guided-journey: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-024] hindu-deity: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-025] image-to-video: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-026] islamic-geometric: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-027] julia: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-028] lecture-scene: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-029] lorenz-attractor: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-030] mandala-3d: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-031] mandala-animated: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-032] mandala-generator: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-033] mandala-personalized: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-034] mandelbrot: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-035] meditation-scene: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-036] nft-art: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-037] print-ready: certify schema validity, resolver mapping, executor
      success, RunPod success, web app run path, and E2E regression.
- [x] [WF-038] rarity-enhancer: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-039] shamanic-vision: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-040] spiritual-teacher: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-041] stable-zero123: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-042] style-transfer: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-043] upscale-enhance: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-044] visionary-art: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.
- [x] [WF-045] yantra-generator: certify schema validity, resolver mapping,
      executor success, RunPod success, web app run path, and E2E regression.

## Workstream 21: Interaction and User-Journey Coverage (No Interaction Missed)

- [x] [UX-001] Anonymous user journey: confirm blocked access to protected
      features with clear recovery path.
- [x] [UX-002] Authenticated job create -> monitor -> cancel journey works for
      every generation type.
- [x] [UX-003] Authenticated workflow authoring journey works (create ->
      validate -> version -> run -> deprecate).
- [x] [UX-004] Authenticated output journey works (discover -> provenance
      inspect -> lineage traverse -> download/restore).
- [x] [UX-005] Model management journey works (register -> upload -> complete ->
      use in workflow -> archive/delete).
- [x] [UX-006] Provider failover journey is visible and understandable to
      users/operators.
- [x] [UX-007] RunPod quota/cost warning journey is visible before expensive
      runs.
- [x] [UX-008] Failure recovery journey works (retry, clone config, report
      issue, support context bundle).
- [x] [UX-009] Cross-project/tenant boundaries are enforced in every user
      journey.
- [x] [UX-010] Every UI interaction has corresponding API route test and
      telemetry event.

## Workstream 22: Operational Resilience and Compliance Completion

- [x] [OPS-001] Add database backup/restore runbooks and recovery time
      objectives for Isis data stores.
- [x] [OPS-002] Add secrets rotation procedures for RunPod API keys and webhook
      secrets.
- [x] [OPS-003] Add disaster recovery drill for provider outage (RunPod
      unavailable) with fallback verification.
- [x] [OPS-004] Add budget guardrails and automatic throttling when RunPod spend
      thresholds are crossed.
- [x] [OPS-005] Add data retention and deletion compliance checks for outputs,
      provenance, and model artifacts.
- [x] [OPS-006] Add post-incident review template specific to generation
      failures and cost overruns.

## Web App Feature Parity Checklist (No Feature Missed)

- [x] [PARITY-001] Every method in `GenerationResource` is accessible via UI
      flow.
- [x] [PARITY-002] Every method in `WorkflowsResource` is accessible via UI
      flow.
- [x] [PARITY-003] Every method in `OutputsResource` is accessible via UI flow.
- [x] [PARITY-004] Every model management endpoint is accessible via UI flow.
- [x] [PARITY-005] Every RunPod operations endpoint is accessible via UI or
      admin UI.
- [x] [PARITY-006] Every long-running action has user-visible progress state.
- [x] [PARITY-007] Every destructive action has confirmation + undo/restore path
      where applicable.
- [x] [PARITY-008] Every page has empty/loading/error states.
- [x] [PARITY-009] Every UI action emits analytics/audit events.

## Mandatory Exit Gates

- [x] [GATE-001] End-to-end production-like workflow run passes for each catalog
      category: `3d`, `avatar`, `fractal`, `nft`, `sacred-art`, `texture`,
      `utility`, `video`.
- [x] [GATE-002] 45/45 workflow definitions pass canonical schema and registry
      validation.
- [x] [GATE-003] No simulated output generation is reachable in production
      environment.
- [x] [GATE-004] RunPod cost and history data are complete and queryable per
      job.
- [x] [GATE-005] Isis web app demonstrates full feature parity in UAT checklist.
- [x] [GATE-006] Security review complete with no critical/high unresolved
      findings.
- [x] [GATE-007] Observability dashboards and alerts are live and validated.
- [x] [GATE-008] Regression suite is green on CI for backend, worker, infra, and
      web.

## Notes

- This checklist is intentionally exhaustive and should be tracked as executable
  engineering work, not documentation-only tasks.
- If any item is descoped, record rationale in an ADR and update risk acceptance
  explicitly.
