Historical task source. The Eve task board holds current task state.
Last Updated: 2026-02-21 Source Audit:
docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md
Objective#
Close every Isis domain gap identified in the audit and deliver production-grade end-to-end workflow execution, RunPod pay-per-second serverless operations, and complete Isis web app feature parity.
Global Definition of Done#
- All P0, P1, and P2 findings from the audit are closed with linked PRs.
- No simulated/placeholder runtime behavior remains in production profiles.
- API, SDK, worker, and web app contracts are aligned and versioned.
- Full observability exists for job lifecycle, cost, latency, and failure modes.
- CI gates block regressions on schema compatibility, execution path, and provider integrations.
- All 45 workflow IDs are certified end-to-end (schema -> execution -> RunPod -> web app -> E2E).
Workstream 00: Program Governance and Delivery Controls#
- [WS00-001] Create an Isis remediation epic with one ticket per checklist item.
- [WS00-002] Assign accountable owner for each finding (F01-F22).
- [WS00-003] Define target milestones for Phase 0/1/2/3 with explicit dates.
- [WS00-004] Create a risk board for execution, infra, and cost regressions.
- [WS00-005] Add architecture decision records (ADRs) for queueing model, schema canonicalization, and RunPod strategy.
- [WS00-006] Define rollback criteria for all production rollouts.
- [WS00-007] Define release quality bar and sign-off matrix (backend, platform, frontend, SRE, security).
- [WS00-008] Add weekly status report template with burn-down by finding.
Workstream 01 (F01): Job Submission to Execution Pipeline (P0)#
- [F01-001] Define canonical job envelope schema in a shared package
(
jobId,type,input,priority,metadata,idempotencyKey). - [F01-002] Add BullMQ queue producer module to generation API.
- [F01-003] Publish queue name and redis config as strict required env vars for production.
- [F01-004] Enqueue on submit and persist status transition to
queued. - [F01-005] Add transactional outbox pattern so DB write and enqueue cannot diverge.
- [F01-006] Add idempotency key handling to prevent duplicate enqueues.
- [F01-007] Ensure cancellation removes queued jobs and marks cancelled reason.
- [F01-008] Add retry policy with capped attempts and exponential backoff.
- [F01-009] Add dead-letter queue with retention policy and replay tooling.
- [F01-010] Emit lifecycle events for queued/running/progress/completed/failed/cancelled from real execution path.
- [F01-011] Ensure worker ACK/NACK semantics update API-side job status deterministically.
- [F01-012] Add queue position endpoint backed by queue state, not only DB estimation.
- [F01-013] Add queue drain/health admin operation with auth guard.
- [F01-014] Add migration/cleanup for legacy
pendingjobs stuck before queue wiring. - [F01-015] Add load tests for enqueue throughput and worker consumption.
Workstream 02 (F02): Workflow Catalog Runtime Integration (P0)#
- [F02-001] Add startup bootstrap to register all entries from
@isis/workflowsinto executor resolver. - [F02-002] Remove hardcoded executor default IDs not present in registry.
- [F02-003] Map each generation type to valid catalog workflow IDs.
- [F02-004] Add validation to fail startup if mapped defaults are missing.
- [F02-005] Add runtime guard to fail closed on unresolved workflow in production.
- [F02-006] Add per-workflow capability metadata (input schema, output schema, provider compatibility).
- [F02-007] Add deterministic workflow selection policy (user override > project default > system default).
- [F02-008] Add workflow cache warm-up at service boot.
- [F02-009] Add resolver metrics (
resolve_success,resolve_failure,fallback_used). - [F02-010] Remove placeholder output URL generation in executor production path.
Workstream 03 (F03): Canonical Workflow Schema and Interop (P0)#
- [F03-001] Choose canonical Isis workflow format (object-map nodes vs array nodes/links).
- [F03-002] Publish schema v1.0 with strict compatibility rules.
- [F03-003] Refactor workflow-registry validator to support canonical schema.
- [F03-004] If dual support is required, implement reversible adapter with lossless conversion tests.
- [F03-005] Migrate all 45 workflows to canonical format if needed.
- [F03-006] Add contract test: every workflow in
libs/isis/workflows/src/workflowspasses registry validation. - [F03-007] Add contract test: registry-exported workflow can execute via executor without schema mutation errors.
- [F03-008] Add schema lint rule in CI to block non-canonical workflow files.
- [F03-009] Add schema version field and migration registry.
- [F03-010] Add docs for custom node contracts and required fields.
Workstream 04 (F04): Isis Web App Full Feature Exposure (P0)#
- [F04-001] Create
apps/isis/webapplication scaffold with routing, auth, and API client wiring. - [F04-002] Add tenant-aware authentication and RBAC.
- [F04-003] Add jobs dashboard (list, filter, sort, status, cancel, retry, delete).
- [F04-004] Add job submission studio supporting all generation types.
- [F04-005] Add workflow catalog page (search/filter/category/tag/GPU requirements).
- [F04-006] Add workflow details page (metadata, schema, node graph preview).
- [F04-007] Add workflow create/edit UI with validation feedback.
- [F04-008] Add workflow version management UI (create/list/get/deprecate/activate).
- [F04-009] Add templates page and create-from-template flow.
- [F04-010] Add workflow run UI with parameter forms generated from schema.
- [F04-011] Add outputs registry page (list/get/update/delete/batch-delete).
- [F04-012] Add output storage operations UI (download URL, upload URL, move tier, restore).
- [F04-013] Add provenance editor and visualization.
- [F04-014] Add lineage graph explorer (ancestors/descendants/both, depth controls).
- [F04-015] Add retention policy management UI.
- [F04-016] Add models page (register, upload complete, list, update, delete, download).
- [F04-017] Add provider operations page (queue stats, health, failover status).
- [F04-018] Add RunPod operations page (cost, usage, active endpoints, failures).
- [F04-019] Add live job progress updates via polling or SSE.
- [F04-020] Add complete audit log panel for workflow and model changes.
- [F04-021] Add feature flags panel (read-only in prod) for provider routing diagnostics.
- [F04-022] Add error state UX for all failure categories (validation/network/provider/auth).
- [F04-023] Add empty states, skeleton loading states, and retry actions across all views.
- [F04-024] Add mobile-responsive layouts for core operational pages.
- [F04-025] Add accessibility compliance checks (keyboard navigation, aria labels, contrast).
Workstream 05 (F05): RunPod Adapter Productionization (P1)#
- [F05-001] Implement real
estimateCost()using GPU pricing + measured runtime estimates. - [F05-002] Implement real
getCredits()integration path (or internal wallet abstraction with source-of-truth sync). - [F05-003] Implement persistent
getJobHistory()with indexed query support. - [F05-004] Implement model inventory endpoint for baked endpoint images.
- [F05-005] Implement custom node inventory endpoint for baked endpoint images.
- [F05-006] Replace synthetic timestamps with provider-sourced timestamps where available.
- [F05-007] Add status mapping contract tests for every RunPod status variant.
- [F05-008] Add retry/circuit-breaker tests for rate limit, timeout, auth failure, and server error.
- [F05-009] Add provider error taxonomy mapping to user-safe messages and machine-readable codes.
- [F05-010] Add tracing context propagation (
correlationId,projectId,userId).
Workstream 06 (F06): RunPod Handler Capability Expansion (P1)#
- [F06-001] Add workflow-id driven execution mode for catalog-native invocation.
- [F06-002] Add endpoint that returns supported workflow IDs and metadata.
- [F06-003] Add typed contracts for major Isis categories (video, 3d, fractal, avatar, sacred-art, utility).
- [F06-004] Add schema validation for workflow overrides and input images per workflow.
- [F06-005] Add compatibility checks for required custom nodes/models before execution.
- [F06-006] Add deterministic error response format for unsupported workflow/category.
- [F06-007] Add execution limits per workflow class (timeout/memory/frame/steps).
- [F06-008] Add test fixtures for each major category.
Workstream 07 (F07): Remove Simulated Runtime Behavior from Executors (P1)#
- [F07-001] Add production guard: simulated paths must be disabled in production builds.
- [F07-002] Texture Upscale: integrate real model inference backend and output validation.
- [F07-003] Gaussian Splatting: integrate real reconstruction pipeline and real output serializers.
- [F07-004] Blender Render: enforce Blender dependency check at startup and fail fast if missing.
- [F07-005] Mesh Processing: replace simulated operations with real toolchain operations.
- [F07-006] Add golden-output tests for each executor.
- [F07-007] Add output checksum and file integrity verification after execution.
- [F07-008] Add performance benchmarks and min throughput targets per executor.
- [F07-009] Add cancellation tests for long-running operations.
- [F07-010] Add resource leak tests (VRAM, temp files, subprocess cleanup).
Workstream 08 (F08): RunPod Provisioning and Drift Control (P1)#
- [F08-001] Decide provisioning mechanism (Terraform provider/external module/API-driven CI job).
- [F08-002] Implement endpoint lifecycle automation (create/update/delete).
- [F08-003] Store endpoint desired state in code (GPU type, min/max workers, idle timeout, execution timeout).
- [F08-004] Add drift detection comparing desired state vs actual endpoint config.
- [F08-005] Add reconcile command for drift remediation.
- [F08-006] Add CI validation for runpod endpoint definitions before deploy.
- [F08-007] Add staged rollout by endpoint with canary traffic percentages.
- [F08-008] Add on-call runbook for endpoint outages and cost spikes.
Workstream 09 (F09): Workflow Registry Persistence and Auth Hardening (P2)#
- [F09-001] Replace in-memory storage with PostgreSQL persistence.
- [F09-002] Add migrations for workflows, versions, templates, stars, runs, and ownership tables.
- [F09-003] Remove
DEFAULT_OWNER = 'user_demo'and source owner from auth claims. - [F09-004] Add RBAC checks for create/update/delete/version operations.
- [F09-005] Add optimistic locking/version conflict handling on updates.
- [F09-006] Add pagination + indexing for large catalogs.
- [F09-007] Add soft-delete + restore support for workflows and versions.
- [F09-008] Add audit trail for all write operations.
Workstream 10 (F10): Output Registry Ownership and Access Control (P2)#
- [F10-001] Remove
DEFAULT_OWNER = 'user_demo'and enforce authenticated owner context. - [F10-002] Add row-level authorization checks for outputs/provenance/lineage.
- [F10-003] Add tenant scoping in all list and graph queries.
- [F10-004] Add policy enforcement for cross-project lineage links.
- [F10-005] Add audit logging for provenance and lineage mutations.
Workstream 11 (F11): Health and Readiness Truthfulness (P2)#
- [F11-001] Replace placeholder readiness values for queue/storage with real checks.
- [F11-002] Add Redis connectivity and queue liveness probe.
- [F11-003] Add object storage read/write probe with timeout and retries.
- [F11-004] Add provider health check with dependency-specific status details.
- [F11-005] Ensure readiness endpoint returns
503on critical dependency failure. - [F11-006] Add alerting based on readiness degradation.
Workstream 12 (F12): Model Storage Presign Productionization (P2)#
- [F12-001] Integrate real pre-signed upload URL generation via
@oshun/storage. - [F12-002] Integrate real pre-signed download URL generation via
@oshun/storage. - [F12-003] Add content hash verification step on
completeUpload. - [F12-004] Enforce MIME/type and size constraints at upload and completion.
- [F12-005] Add anti-virus/malware scan hook for uploaded model files.
- [F12-006] Add retention/cleanup policy for abandoned uploads.
Workstream 13 (F13): Deployment Configuration Drift (P2)#
- [F13-001] Update docker compose Isis app paths to
apps/isis/generation-apiandapps/isis/gpu-worker. - [F13-002] Add CI check to validate
APP_PATHvalues exist for all compose services. - [F13-003] Add smoke test that boots Isis services via compose and verifies health endpoints.
- [F13-004] Document local startup workflow for Isis services and GPU profile.
Workstream 14 (F14): Test Coverage Expansion for Real Execution (P2)#
- [F14-001] Add full E2E test: submit job -> queue -> worker -> output -> completed state.
- [F14-002] Add E2E test for cancel during queued and running states.
- [F14-003] Add E2E test for retry flow with bounded attempts.
- [F14-004] Add contract test for all 45 workflows executing validation + resolution.
- [F14-005] Add RunPod sandbox integration tests for submit/status/cancel.
- [F14-006] Add cost-accounting correctness tests with fixture runtimes.
- [F14-007] Add web app E2E tests covering every major interaction surface.
- [F14-008] Add chaos tests for provider failover and queue outage handling.
Workstream 15 (F15): API and SDK Contract Parity (Critical Completeness)#
- [F15-001] Inventory all
@isis/clientmethods and map to live API routes. - [F15-002] Resolve generation route mismatches (
/api/v1/jobs/batch,/api/v1/jobs/:id/retry,/api/v1/queue/stats,/api/v1/jobs/:id/queue-position). - [F15-003] Resolve workflows route mismatches (metadata/clone/activate/tag/version validation endpoints).
- [F15-004] Resolve outputs route mismatches (batch-delete, download/upload-url, move-tier, restore, provenance update, policy endpoints).
- [F15-005] Decide whether to add missing endpoints or deprecate unsupported SDK methods.
- [F15-006] Add OpenAPI specs for generation/workflow/output services.
- [F15-007] Generate SDK from OpenAPI or add CI contract tests to prevent drift.
- [F15-008] Add breaking-change detection job in CI for API route contracts.
Workstream 16: Security and Multi-Tenant Hardening#
- [SEC-001] Add auth middleware consistency across generation, workflow, output, and model endpoints.
- [SEC-002] Enforce project-level authorization on every read/write route.
- [SEC-003] Add API key scopes for admin/ops/provider-control actions.
- [SEC-004] Add webhook signature verification for provider callbacks.
- [SEC-005] Add input sanitization and payload size limits on workflow definitions.
- [SEC-006] Add structured security logs for sensitive operations.
- [SEC-007] Add abuse controls for expensive generation endpoints.
Workstream 17: Observability, Cost, and SLOs#
- [OBS-001] Define SLOs for submit latency, queue delay, execution success rate, and end-to-end completion time.
- [OBS-002] Add metrics for queue depth, stuck jobs, execution retries, and DLQ size.
- [OBS-003] Add per-provider latency and failure dashboards.
- [OBS-004] Add cost dashboards by endpoint, project, workflow, and user.
- [OBS-005] Add anomaly alerts for cost spikes and failure rates.
- [OBS-006] Add distributed tracing across API -> queue -> worker -> provider -> storage.
- [OBS-007] Add operator CLI commands for stuck-job triage and replay.
Workstream 18: Data Migration and Backfill#
- [MIG-001] Design migration for existing jobs to new lifecycle/status model.
- [MIG-002] Backfill workflow IDs for historical jobs where missing.
- [MIG-003] Backfill provenance records for completed outputs where possible.
- [MIG-004] Backfill cost data from historical provider logs.
- [MIG-005] Add migration rollback and verification scripts.
Workstream 19: Release Plan and Cutover#
- [REL-001] Enable queue producer behind feature flag for safe rollout.
- [REL-002] Canary release to limited traffic/project set.
- [REL-003] Verify SLO and cost behavior during canary.
- [REL-004] Gradually increase rollout percentage with auto-rollback thresholds.
- [REL-005] Conduct game-day exercises before full cutover.
- [REL-006] Finalize post-cutover validation checklist and sign-off.
Workstream 20: Per-Workflow End-to-End Certification (45/45)#
- [WF-001] 3d-texture: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-002] animatediff-controlnet: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-003] artifact-reconstruction: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-004] background-removal: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-005] barnsley-fern: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-006] base-avatar: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-007] batch-resize: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-008] batch-watermark: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-009] buddhist-thangka: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-010] celtic-knotwork: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-011] christian-icon: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-012] concept-explainer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-013] consciousness-map: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-014] consistent-character: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-015] cosmetic-layer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-016] egyptian-hieratic: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-017] environment-map: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-018] face-swap: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-019] flame-fractal: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-020] fractal-art-fusion: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-021] fractal-morph: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-022] fractal-zoom: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-023] guided-journey: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-024] hindu-deity: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-025] image-to-video: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-026] islamic-geometric: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-027] julia: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-028] lecture-scene: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-029] lorenz-attractor: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-030] mandala-3d: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-031] mandala-animated: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-032] mandala-generator: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-033] mandala-personalized: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-034] mandelbrot: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-035] meditation-scene: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-036] nft-art: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-037] print-ready: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-038] rarity-enhancer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-039] shamanic-vision: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-040] spiritual-teacher: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-041] stable-zero123: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-042] style-transfer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-043] upscale-enhance: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-044] visionary-art: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
- [WF-045] yantra-generator: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
Workstream 21: Interaction and User-Journey Coverage (No Interaction Missed)#
- [UX-001] Anonymous user journey: confirm blocked access to protected features with clear recovery path.
- [UX-002] Authenticated job create -> monitor -> cancel journey works for every generation type.
- [UX-003] Authenticated workflow authoring journey works (create -> validate -> version -> run -> deprecate).
- [UX-004] Authenticated output journey works (discover -> provenance inspect -> lineage traverse -> download/restore).
- [UX-005] Model management journey works (register -> upload -> complete -> use in workflow -> archive/delete).
- [UX-006] Provider failover journey is visible and understandable to users/operators.
- [UX-007] RunPod quota/cost warning journey is visible before expensive runs.
- [UX-008] Failure recovery journey works (retry, clone config, report issue, support context bundle).
- [UX-009] Cross-project/tenant boundaries are enforced in every user journey.
- [UX-010] Every UI interaction has corresponding API route test and telemetry event.
Workstream 22: Operational Resilience and Compliance Completion#
- [OPS-001] Add database backup/restore runbooks and recovery time objectives for Isis data stores.
- [OPS-002] Add secrets rotation procedures for RunPod API keys and webhook secrets.
- [OPS-003] Add disaster recovery drill for provider outage (RunPod unavailable) with fallback verification.
- [OPS-004] Add budget guardrails and automatic throttling when RunPod spend thresholds are crossed.
- [OPS-005] Add data retention and deletion compliance checks for outputs, provenance, and model artifacts.
- [OPS-006] Add post-incident review template specific to generation failures and cost overruns.
Web App Feature Parity Checklist (No Feature Missed)#
- [PARITY-001] Every method in
GenerationResourceis accessible via UI flow. - [PARITY-002] Every method in
WorkflowsResourceis accessible via UI flow. - [PARITY-003] Every method in
OutputsResourceis accessible via UI flow. - [PARITY-004] Every model management endpoint is accessible via UI flow.
- [PARITY-005] Every RunPod operations endpoint is accessible via UI or admin UI.
- [PARITY-006] Every long-running action has user-visible progress state.
- [PARITY-007] Every destructive action has confirmation + undo/restore path where applicable.
- [PARITY-008] Every page has empty/loading/error states.
- [PARITY-009] Every UI action emits analytics/audit events.
Mandatory Exit Gates#
- [GATE-001] End-to-end production-like workflow run passes for each catalog
category:
3d,avatar,fractal,nft,sacred-art,texture,utility,video. - [GATE-002] 45/45 workflow definitions pass canonical schema and registry validation.
- [GATE-003] No simulated output generation is reachable in production environment.
- [GATE-004] RunPod cost and history data are complete and queryable per job.
- [GATE-005] Isis web app demonstrates full feature parity in UAT checklist.
- [GATE-006] Security review complete with no critical/high unresolved findings.
- [GATE-007] Observability dashboards and alerts are live and validated.
- [GATE-008] Regression suite is green on CI for backend, worker, infra, and web.
Notes#
- This checklist is intentionally exhaustive and should be tracked as executable engineering work, not documentation-only tasks.
- If any item is descoped, record rationale in an ADR and update risk acceptance explicitly.