Eve library

ISIS_TODOS.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Last Updated: 2026-02-21 Source Audit: docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md

Objective#

Close every Isis domain gap identified in the audit and deliver production-grade end-to-end workflow execution, RunPod pay-per-second serverless operations, and complete Isis web app feature parity.

Global Definition of Done#

  • All P0, P1, and P2 findings from the audit are closed with linked PRs.
  • No simulated/placeholder runtime behavior remains in production profiles.
  • API, SDK, worker, and web app contracts are aligned and versioned.
  • Full observability exists for job lifecycle, cost, latency, and failure modes.
  • CI gates block regressions on schema compatibility, execution path, and provider integrations.
  • All 45 workflow IDs are certified end-to-end (schema -> execution -> RunPod -> web app -> E2E).

Workstream 00: Program Governance and Delivery Controls#

  • [WS00-001] Create an Isis remediation epic with one ticket per checklist item.
  • [WS00-002] Assign accountable owner for each finding (F01-F22).
  • [WS00-003] Define target milestones for Phase 0/1/2/3 with explicit dates.
  • [WS00-004] Create a risk board for execution, infra, and cost regressions.
  • [WS00-005] Add architecture decision records (ADRs) for queueing model, schema canonicalization, and RunPod strategy.
  • [WS00-006] Define rollback criteria for all production rollouts.
  • [WS00-007] Define release quality bar and sign-off matrix (backend, platform, frontend, SRE, security).
  • [WS00-008] Add weekly status report template with burn-down by finding.

Workstream 01 (F01): Job Submission to Execution Pipeline (P0)#

  • [F01-001] Define canonical job envelope schema in a shared package (jobId, type, input, priority, metadata, idempotencyKey).
  • [F01-002] Add BullMQ queue producer module to generation API.
  • [F01-003] Publish queue name and redis config as strict required env vars for production.
  • [F01-004] Enqueue on submit and persist status transition to queued.
  • [F01-005] Add transactional outbox pattern so DB write and enqueue cannot diverge.
  • [F01-006] Add idempotency key handling to prevent duplicate enqueues.
  • [F01-007] Ensure cancellation removes queued jobs and marks cancelled reason.
  • [F01-008] Add retry policy with capped attempts and exponential backoff.
  • [F01-009] Add dead-letter queue with retention policy and replay tooling.
  • [F01-010] Emit lifecycle events for queued/running/progress/completed/failed/cancelled from real execution path.
  • [F01-011] Ensure worker ACK/NACK semantics update API-side job status deterministically.
  • [F01-012] Add queue position endpoint backed by queue state, not only DB estimation.
  • [F01-013] Add queue drain/health admin operation with auth guard.
  • [F01-014] Add migration/cleanup for legacy pending jobs stuck before queue wiring.
  • [F01-015] Add load tests for enqueue throughput and worker consumption.

Workstream 02 (F02): Workflow Catalog Runtime Integration (P0)#

  • [F02-001] Add startup bootstrap to register all entries from @isis/workflows into executor resolver.
  • [F02-002] Remove hardcoded executor default IDs not present in registry.
  • [F02-003] Map each generation type to valid catalog workflow IDs.
  • [F02-004] Add validation to fail startup if mapped defaults are missing.
  • [F02-005] Add runtime guard to fail closed on unresolved workflow in production.
  • [F02-006] Add per-workflow capability metadata (input schema, output schema, provider compatibility).
  • [F02-007] Add deterministic workflow selection policy (user override > project default > system default).
  • [F02-008] Add workflow cache warm-up at service boot.
  • [F02-009] Add resolver metrics (resolve_success, resolve_failure, fallback_used).
  • [F02-010] Remove placeholder output URL generation in executor production path.

Workstream 03 (F03): Canonical Workflow Schema and Interop (P0)#

  • [F03-001] Choose canonical Isis workflow format (object-map nodes vs array nodes/links).
  • [F03-002] Publish schema v1.0 with strict compatibility rules.
  • [F03-003] Refactor workflow-registry validator to support canonical schema.
  • [F03-004] If dual support is required, implement reversible adapter with lossless conversion tests.
  • [F03-005] Migrate all 45 workflows to canonical format if needed.
  • [F03-006] Add contract test: every workflow in libs/isis/workflows/src/workflows passes registry validation.
  • [F03-007] Add contract test: registry-exported workflow can execute via executor without schema mutation errors.
  • [F03-008] Add schema lint rule in CI to block non-canonical workflow files.
  • [F03-009] Add schema version field and migration registry.
  • [F03-010] Add docs for custom node contracts and required fields.

Workstream 04 (F04): Isis Web App Full Feature Exposure (P0)#

  • [F04-001] Create apps/isis/web application scaffold with routing, auth, and API client wiring.
  • [F04-002] Add tenant-aware authentication and RBAC.
  • [F04-003] Add jobs dashboard (list, filter, sort, status, cancel, retry, delete).
  • [F04-004] Add job submission studio supporting all generation types.
  • [F04-005] Add workflow catalog page (search/filter/category/tag/GPU requirements).
  • [F04-006] Add workflow details page (metadata, schema, node graph preview).
  • [F04-007] Add workflow create/edit UI with validation feedback.
  • [F04-008] Add workflow version management UI (create/list/get/deprecate/activate).
  • [F04-009] Add templates page and create-from-template flow.
  • [F04-010] Add workflow run UI with parameter forms generated from schema.
  • [F04-011] Add outputs registry page (list/get/update/delete/batch-delete).
  • [F04-012] Add output storage operations UI (download URL, upload URL, move tier, restore).
  • [F04-013] Add provenance editor and visualization.
  • [F04-014] Add lineage graph explorer (ancestors/descendants/both, depth controls).
  • [F04-015] Add retention policy management UI.
  • [F04-016] Add models page (register, upload complete, list, update, delete, download).
  • [F04-017] Add provider operations page (queue stats, health, failover status).
  • [F04-018] Add RunPod operations page (cost, usage, active endpoints, failures).
  • [F04-019] Add live job progress updates via polling or SSE.
  • [F04-020] Add complete audit log panel for workflow and model changes.
  • [F04-021] Add feature flags panel (read-only in prod) for provider routing diagnostics.
  • [F04-022] Add error state UX for all failure categories (validation/network/provider/auth).
  • [F04-023] Add empty states, skeleton loading states, and retry actions across all views.
  • [F04-024] Add mobile-responsive layouts for core operational pages.
  • [F04-025] Add accessibility compliance checks (keyboard navigation, aria labels, contrast).

Workstream 05 (F05): RunPod Adapter Productionization (P1)#

  • [F05-001] Implement real estimateCost() using GPU pricing + measured runtime estimates.
  • [F05-002] Implement real getCredits() integration path (or internal wallet abstraction with source-of-truth sync).
  • [F05-003] Implement persistent getJobHistory() with indexed query support.
  • [F05-004] Implement model inventory endpoint for baked endpoint images.
  • [F05-005] Implement custom node inventory endpoint for baked endpoint images.
  • [F05-006] Replace synthetic timestamps with provider-sourced timestamps where available.
  • [F05-007] Add status mapping contract tests for every RunPod status variant.
  • [F05-008] Add retry/circuit-breaker tests for rate limit, timeout, auth failure, and server error.
  • [F05-009] Add provider error taxonomy mapping to user-safe messages and machine-readable codes.
  • [F05-010] Add tracing context propagation (correlationId, projectId, userId).

Workstream 06 (F06): RunPod Handler Capability Expansion (P1)#

  • [F06-001] Add workflow-id driven execution mode for catalog-native invocation.
  • [F06-002] Add endpoint that returns supported workflow IDs and metadata.
  • [F06-003] Add typed contracts for major Isis categories (video, 3d, fractal, avatar, sacred-art, utility).
  • [F06-004] Add schema validation for workflow overrides and input images per workflow.
  • [F06-005] Add compatibility checks for required custom nodes/models before execution.
  • [F06-006] Add deterministic error response format for unsupported workflow/category.
  • [F06-007] Add execution limits per workflow class (timeout/memory/frame/steps).
  • [F06-008] Add test fixtures for each major category.

Workstream 07 (F07): Remove Simulated Runtime Behavior from Executors (P1)#

  • [F07-001] Add production guard: simulated paths must be disabled in production builds.
  • [F07-002] Texture Upscale: integrate real model inference backend and output validation.
  • [F07-003] Gaussian Splatting: integrate real reconstruction pipeline and real output serializers.
  • [F07-004] Blender Render: enforce Blender dependency check at startup and fail fast if missing.
  • [F07-005] Mesh Processing: replace simulated operations with real toolchain operations.
  • [F07-006] Add golden-output tests for each executor.
  • [F07-007] Add output checksum and file integrity verification after execution.
  • [F07-008] Add performance benchmarks and min throughput targets per executor.
  • [F07-009] Add cancellation tests for long-running operations.
  • [F07-010] Add resource leak tests (VRAM, temp files, subprocess cleanup).

Workstream 08 (F08): RunPod Provisioning and Drift Control (P1)#

  • [F08-001] Decide provisioning mechanism (Terraform provider/external module/API-driven CI job).
  • [F08-002] Implement endpoint lifecycle automation (create/update/delete).
  • [F08-003] Store endpoint desired state in code (GPU type, min/max workers, idle timeout, execution timeout).
  • [F08-004] Add drift detection comparing desired state vs actual endpoint config.
  • [F08-005] Add reconcile command for drift remediation.
  • [F08-006] Add CI validation for runpod endpoint definitions before deploy.
  • [F08-007] Add staged rollout by endpoint with canary traffic percentages.
  • [F08-008] Add on-call runbook for endpoint outages and cost spikes.

Workstream 09 (F09): Workflow Registry Persistence and Auth Hardening (P2)#

  • [F09-001] Replace in-memory storage with PostgreSQL persistence.
  • [F09-002] Add migrations for workflows, versions, templates, stars, runs, and ownership tables.
  • [F09-003] Remove DEFAULT_OWNER = 'user_demo' and source owner from auth claims.
  • [F09-004] Add RBAC checks for create/update/delete/version operations.
  • [F09-005] Add optimistic locking/version conflict handling on updates.
  • [F09-006] Add pagination + indexing for large catalogs.
  • [F09-007] Add soft-delete + restore support for workflows and versions.
  • [F09-008] Add audit trail for all write operations.

Workstream 10 (F10): Output Registry Ownership and Access Control (P2)#

  • [F10-001] Remove DEFAULT_OWNER = 'user_demo' and enforce authenticated owner context.
  • [F10-002] Add row-level authorization checks for outputs/provenance/lineage.
  • [F10-003] Add tenant scoping in all list and graph queries.
  • [F10-004] Add policy enforcement for cross-project lineage links.
  • [F10-005] Add audit logging for provenance and lineage mutations.

Workstream 11 (F11): Health and Readiness Truthfulness (P2)#

  • [F11-001] Replace placeholder readiness values for queue/storage with real checks.
  • [F11-002] Add Redis connectivity and queue liveness probe.
  • [F11-003] Add object storage read/write probe with timeout and retries.
  • [F11-004] Add provider health check with dependency-specific status details.
  • [F11-005] Ensure readiness endpoint returns 503 on critical dependency failure.
  • [F11-006] Add alerting based on readiness degradation.

Workstream 12 (F12): Model Storage Presign Productionization (P2)#

  • [F12-001] Integrate real pre-signed upload URL generation via @oshun/storage.
  • [F12-002] Integrate real pre-signed download URL generation via @oshun/storage.
  • [F12-003] Add content hash verification step on completeUpload.
  • [F12-004] Enforce MIME/type and size constraints at upload and completion.
  • [F12-005] Add anti-virus/malware scan hook for uploaded model files.
  • [F12-006] Add retention/cleanup policy for abandoned uploads.

Workstream 13 (F13): Deployment Configuration Drift (P2)#

  • [F13-001] Update docker compose Isis app paths to apps/isis/generation-api and apps/isis/gpu-worker.
  • [F13-002] Add CI check to validate APP_PATH values exist for all compose services.
  • [F13-003] Add smoke test that boots Isis services via compose and verifies health endpoints.
  • [F13-004] Document local startup workflow for Isis services and GPU profile.

Workstream 14 (F14): Test Coverage Expansion for Real Execution (P2)#

  • [F14-001] Add full E2E test: submit job -> queue -> worker -> output -> completed state.
  • [F14-002] Add E2E test for cancel during queued and running states.
  • [F14-003] Add E2E test for retry flow with bounded attempts.
  • [F14-004] Add contract test for all 45 workflows executing validation + resolution.
  • [F14-005] Add RunPod sandbox integration tests for submit/status/cancel.
  • [F14-006] Add cost-accounting correctness tests with fixture runtimes.
  • [F14-007] Add web app E2E tests covering every major interaction surface.
  • [F14-008] Add chaos tests for provider failover and queue outage handling.

Workstream 15 (F15): API and SDK Contract Parity (Critical Completeness)#

  • [F15-001] Inventory all @isis/client methods and map to live API routes.
  • [F15-002] Resolve generation route mismatches (/api/v1/jobs/batch, /api/v1/jobs/:id/retry, /api/v1/queue/stats, /api/v1/jobs/:id/queue-position).
  • [F15-003] Resolve workflows route mismatches (metadata/clone/activate/tag/version validation endpoints).
  • [F15-004] Resolve outputs route mismatches (batch-delete, download/upload-url, move-tier, restore, provenance update, policy endpoints).
  • [F15-005] Decide whether to add missing endpoints or deprecate unsupported SDK methods.
  • [F15-006] Add OpenAPI specs for generation/workflow/output services.
  • [F15-007] Generate SDK from OpenAPI or add CI contract tests to prevent drift.
  • [F15-008] Add breaking-change detection job in CI for API route contracts.

Workstream 16: Security and Multi-Tenant Hardening#

  • [SEC-001] Add auth middleware consistency across generation, workflow, output, and model endpoints.
  • [SEC-002] Enforce project-level authorization on every read/write route.
  • [SEC-003] Add API key scopes for admin/ops/provider-control actions.
  • [SEC-004] Add webhook signature verification for provider callbacks.
  • [SEC-005] Add input sanitization and payload size limits on workflow definitions.
  • [SEC-006] Add structured security logs for sensitive operations.
  • [SEC-007] Add abuse controls for expensive generation endpoints.

Workstream 17: Observability, Cost, and SLOs#

  • [OBS-001] Define SLOs for submit latency, queue delay, execution success rate, and end-to-end completion time.
  • [OBS-002] Add metrics for queue depth, stuck jobs, execution retries, and DLQ size.
  • [OBS-003] Add per-provider latency and failure dashboards.
  • [OBS-004] Add cost dashboards by endpoint, project, workflow, and user.
  • [OBS-005] Add anomaly alerts for cost spikes and failure rates.
  • [OBS-006] Add distributed tracing across API -> queue -> worker -> provider -> storage.
  • [OBS-007] Add operator CLI commands for stuck-job triage and replay.

Workstream 18: Data Migration and Backfill#

  • [MIG-001] Design migration for existing jobs to new lifecycle/status model.
  • [MIG-002] Backfill workflow IDs for historical jobs where missing.
  • [MIG-003] Backfill provenance records for completed outputs where possible.
  • [MIG-004] Backfill cost data from historical provider logs.
  • [MIG-005] Add migration rollback and verification scripts.

Workstream 19: Release Plan and Cutover#

  • [REL-001] Enable queue producer behind feature flag for safe rollout.
  • [REL-002] Canary release to limited traffic/project set.
  • [REL-003] Verify SLO and cost behavior during canary.
  • [REL-004] Gradually increase rollout percentage with auto-rollback thresholds.
  • [REL-005] Conduct game-day exercises before full cutover.
  • [REL-006] Finalize post-cutover validation checklist and sign-off.

Workstream 20: Per-Workflow End-to-End Certification (45/45)#

  • [WF-001] 3d-texture: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-002] animatediff-controlnet: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-003] artifact-reconstruction: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-004] background-removal: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-005] barnsley-fern: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-006] base-avatar: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-007] batch-resize: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-008] batch-watermark: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-009] buddhist-thangka: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-010] celtic-knotwork: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-011] christian-icon: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-012] concept-explainer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-013] consciousness-map: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-014] consistent-character: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-015] cosmetic-layer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-016] egyptian-hieratic: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-017] environment-map: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-018] face-swap: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-019] flame-fractal: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-020] fractal-art-fusion: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-021] fractal-morph: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-022] fractal-zoom: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-023] guided-journey: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-024] hindu-deity: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-025] image-to-video: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-026] islamic-geometric: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-027] julia: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-028] lecture-scene: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-029] lorenz-attractor: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-030] mandala-3d: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-031] mandala-animated: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-032] mandala-generator: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-033] mandala-personalized: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-034] mandelbrot: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-035] meditation-scene: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-036] nft-art: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-037] print-ready: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-038] rarity-enhancer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-039] shamanic-vision: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-040] spiritual-teacher: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-041] stable-zero123: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-042] style-transfer: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-043] upscale-enhance: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-044] visionary-art: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.
  • [WF-045] yantra-generator: certify schema validity, resolver mapping, executor success, RunPod success, web app run path, and E2E regression.

Workstream 21: Interaction and User-Journey Coverage (No Interaction Missed)#

  • [UX-001] Anonymous user journey: confirm blocked access to protected features with clear recovery path.
  • [UX-002] Authenticated job create -> monitor -> cancel journey works for every generation type.
  • [UX-003] Authenticated workflow authoring journey works (create -> validate -> version -> run -> deprecate).
  • [UX-004] Authenticated output journey works (discover -> provenance inspect -> lineage traverse -> download/restore).
  • [UX-005] Model management journey works (register -> upload -> complete -> use in workflow -> archive/delete).
  • [UX-006] Provider failover journey is visible and understandable to users/operators.
  • [UX-007] RunPod quota/cost warning journey is visible before expensive runs.
  • [UX-008] Failure recovery journey works (retry, clone config, report issue, support context bundle).
  • [UX-009] Cross-project/tenant boundaries are enforced in every user journey.
  • [UX-010] Every UI interaction has corresponding API route test and telemetry event.

Workstream 22: Operational Resilience and Compliance Completion#

  • [OPS-001] Add database backup/restore runbooks and recovery time objectives for Isis data stores.
  • [OPS-002] Add secrets rotation procedures for RunPod API keys and webhook secrets.
  • [OPS-003] Add disaster recovery drill for provider outage (RunPod unavailable) with fallback verification.
  • [OPS-004] Add budget guardrails and automatic throttling when RunPod spend thresholds are crossed.
  • [OPS-005] Add data retention and deletion compliance checks for outputs, provenance, and model artifacts.
  • [OPS-006] Add post-incident review template specific to generation failures and cost overruns.

Web App Feature Parity Checklist (No Feature Missed)#

  • [PARITY-001] Every method in GenerationResource is accessible via UI flow.
  • [PARITY-002] Every method in WorkflowsResource is accessible via UI flow.
  • [PARITY-003] Every method in OutputsResource is accessible via UI flow.
  • [PARITY-004] Every model management endpoint is accessible via UI flow.
  • [PARITY-005] Every RunPod operations endpoint is accessible via UI or admin UI.
  • [PARITY-006] Every long-running action has user-visible progress state.
  • [PARITY-007] Every destructive action has confirmation + undo/restore path where applicable.
  • [PARITY-008] Every page has empty/loading/error states.
  • [PARITY-009] Every UI action emits analytics/audit events.

Mandatory Exit Gates#

  • [GATE-001] End-to-end production-like workflow run passes for each catalog category: 3d, avatar, fractal, nft, sacred-art, texture, utility, video.
  • [GATE-002] 45/45 workflow definitions pass canonical schema and registry validation.
  • [GATE-003] No simulated output generation is reachable in production environment.
  • [GATE-004] RunPod cost and history data are complete and queryable per job.
  • [GATE-005] Isis web app demonstrates full feature parity in UAT checklist.
  • [GATE-006] Security review complete with no critical/high unresolved findings.
  • [GATE-007] Observability dashboards and alerts are live and validated.
  • [GATE-008] Regression suite is green on CI for backend, worker, infra, and web.

Notes#

  • This checklist is intentionally exhaustive and should be tracked as executable engineering work, not documentation-only tasks.
  • If any item is descoped, record rationale in an ADR and update risk acceptance explicitly.