Date: 2026-02-21 Reviewer: Codex (static codebase audit)
Scope#
This audit reviewed the Isis domain for three requirements:
- Complex ComfyUI workflow coverage and implementation completeness.
- RunPod integration for pay-per-second serverless execution and exposure of Isis flows.
- Full feature/workflow exposure via an Isis web app.
Primary paths reviewed:
apps/isis/generation-apiapps/isis/workflow-registryapps/isis/output-registryapps/isis/gpu-workerlibs/isis/workflowslibs/isis/ai-providersdocker/runpod/comfyuiinfra/terraformtesting/e2e/tests/api/isistests/integration/flows/isis-yemaya.integration.test.ts
Executive Summary#
The Isis domain is not currently complete against the requested target state.
Key status:
- Workflow catalog exists (45 workflow JSON files across 8 categories), but end-to-end execution of this catalog is not wired.
- Generation API job submission persists and emits events, but does not enqueue or execute jobs.
- RunPod integration exists at adapter and handler layers, but several pay-per-second operational capabilities are stubbed or partially implemented.
- No dedicated Isis web app exists, and frontend apps do not currently import/use
@isis/clientin runtime source.
Readiness assessment (estimated):
- ComfyUI workflow runtime completeness: Low.
- RunPod serverless production readiness: Medium-Low.
- Isis web app feature exposure completeness: Not implemented.
Capability Matrix#
| Capability | Expected | Observed | Status |
|---|---|---|---|
| Complex workflow inventory | Broad category coverage | 45 workflows, 8 categories in libs/isis/workflows/src/workflows |
Partial |
| Catalog execution wiring | Workflow catalog used at runtime | Executor defaults are internal fallback IDs; catalog not wired into executor registration | Missing |
| API submit -> execution | Job submit triggers processing | submit() stores job as pending and returns queue estimate; no queue producer |
Missing |
| Worker processing path | Shared queue between API and worker | GPU worker consumes BullMQ queue (gpu-jobs), but generation API has no BullMQ producer |
Missing |
| RunPod invocation | Serverless execution path available | Adapter exists and is initialized in ComfyUI service | Partial |
| Pay-per-second cost/account insight | Accurate cost/history/credits surfaced | Cost, credits, and history methods are stubbed/empty | Missing |
| RunPod endpoint provisioning IaC | Endpoints created/configured via infra code | Secrets/monitoring present, endpoint creation resources absent | Partial |
| Full Isis web app | UI exposes all features/interactions/workflows | No apps/isis web app; no frontend runtime imports of @isis/client |
Missing |
Findings and Gaps#
P0-1: Job submission is disconnected from execution#
- Evidence:
apps/isis/generation-api/src/services/job.service.ts:508(submitcreatespendingjob and returns status metadata)apps/isis/generation-api/src/services/job.service.ts:536(queue position derived from DB records)apps/isis/generation-api/src/routes/jobs.ts:24(route only callsjobService.submit)apps/isis/gpu-worker/src/workers/gpu-worker.ts:100(worker consumes BullMQ queue)- No BullMQ producer matches in generation API (
rgforbullmq|new Queue|Worker\(found no results inapps/isis/generation-api/src).
- Impact:
- Submitted jobs can remain pending without entering an executable pipeline.
- Remediation:
- Add a queue producer in generation API that enqueues normalized job payloads into the same queue consumed by GPU worker.
- Add deterministic state transitions:
pending -> queued -> running -> completed/failed. - Add idempotent retry and dead-letter queue semantics.
P0-2: Workflow runtime and workflow catalog are not integrated#
- Evidence:
apps/isis/generation-api/src/workflows/executor.ts:371(resolver initialized empty)apps/isis/generation-api/src/workflows/executor.ts:68(defaults use IDs likeflux-dev-basic,wan-basic)libs/isis/workflows/src/utils/registry.ts(45 registered IDs; default executor IDs are not present)apps/isis/generation-api/src/workflows/executor.ts:888(simulated execution path)apps/isis/generation-api/src/workflows/executor.ts:931(placeholder URLstorage.example.com)
- Impact:
- Complex curated workflows are not the runtime source of truth.
- Runtime can degrade to synthetic output behavior.
- Remediation:
- Add bootstrap that registers
libs/isis/workflowscatalog into executor resolver. - Replace hardcoded default IDs with catalog-backed IDs.
- Fail closed (explicit error) if workflow resolution fails in production.
- Add bootstrap that registers
P0-3: Workflow format contract mismatch across Isis components#
- Evidence:
libs/isis/workflows/src/schemas/workflow.schema.ts:217(WorkflowNodesSchemaisz.record, i.e., object map)libs/isis/workflows/src/workflows/3d/artifact-reconstruction.json:214("nodes": { ... }object)apps/isis/workflow-registry/src/services/workflow.service.ts:537(validator requiresdefinition.nodesarray)apps/isis/workflow-registry/src/services/workflow.service.ts:546(validator requiresdefinition.linksarray)
- Impact:
- Catalog workflows and registry validation are structurally incompatible.
- Imported complex workflows can fail validation despite being valid under library schema.
- Remediation:
- Define one canonical workflow schema for Isis.
- Add adapter/conversion layer only if dual formats are intentionally supported.
- Add contract tests to verify all 45 workflows round-trip through registry validation.
P0-4: No Isis web app exposing all interactions/workflows#
- Evidence:
apps/isiscontainscli,generation-api,gpu-worker,output-registry,workflow-registry(no web app directory).rgfound no runtime source imports of@isis/clientin:apps/yemaya/studio-web/srcapps/yemaya/studio-desktop/srcapps/lilith/web/srcapps/oshun/mobile/src
- Impact:
- Requirement “every feature/workflow exposed via Isis web app” is currently unmet.
- Remediation:
- Create
apps/isis/web(or equivalent) and implement full UI surface:- Job submission and lifecycle controls
- Workflow catalog browsing/validation/runs
- Output/provenance/lineage management
- Model registration and upload/download flows
- RunPod provider selection and observability
- Create
P1-5: RunPod adapter has production-critical stubs#
- Evidence:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:784(estimateCostreturns static placeholder)libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:799(getCreditsreturns zeros)libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:812(getJobHistoryreturns empty)libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:752(listModelsreturns empty)libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:761(listCustomNodesreturns empty)libs/isis/ai-providers/src/providers/comfy/runpod-adapter.ts:711(created/started timestamps synthesized vianew Date())
- Impact:
- Pay-per-second governance (cost tracking, accountability, job history) is incomplete.
- Remediation:
- Implement persistent internal accounting from submit/start/complete events.
- Track worker runtime seconds and map to per-GPU pricing table.
- Persist provider job metadata and expose queryable history endpoints.
P1-6: RunPod handler supports limited typed modes; full Isis catalog exposure is not explicit#
- Evidence:
docker/runpod/comfyui/handler.py:176typed input restricted totxt2img,img2img,inpaint,upscale,workflowdocker/runpod/comfyui/handler.py:522-530type validation gates those modes
- Impact:
- Domain-specific Isis categories (fractal/sacred-art/avatar/3d/video variants) are only indirectly accessible through generic
workflowpayloads, with no explicit discovery/exposure contract.
- Domain-specific Isis categories (fractal/sacred-art/avatar/3d/video variants) are only indirectly accessible through generic
- Remediation:
- Add catalog-backed typed endpoints/contracts or explicit workflow-id routing on serverless handler.
- Publish an endpoint capability map and enforce compatibility checks.
P1-7: GPU worker executors include simulated/placeholder behavior#
- Evidence:
apps/isis/gpu-worker/src/executors/texture-upscale-executor.ts:102(simulate process; placeholder output)apps/isis/gpu-worker/src/executors/gaussian-splatting-executor.ts:267(placeholder exports)apps/isis/gpu-worker/src/executors/blender-render-executor.ts:82(mock mode when Blender absent)apps/isis/gpu-worker/src/executors/mesh-processing-executor.ts:224(simulate operations/copy placeholder)
- Impact:
- Several advanced workflows are not backed by deterministic production execution.
- Remediation:
- Gate mock/simulated modes behind explicit non-production flags.
- Implement real toolchain invocation and output validation checksums.
P1-8: Infrastructure covers secrets/monitoring for RunPod, but endpoint lifecycle provisioning is missing#
- Evidence:
infra/terraform/secrets/runpod.tfstores RunPod API and endpoint IDs as secrets.infra/terraform/cloudwatch/dashboards.tfandinfra/terraform/cloudwatch/alarms.tfmonitor RunPod metrics.- No RunPod endpoint creation resources/modules found under
infra/terraform(resource "*runpod"only secrets/monitoring context).
- Impact:
- Endpoint setup remains manual and drift-prone; reproducibility is reduced.
- Remediation:
- Add explicit endpoint provisioning automation (Terraform external provider/module or managed CI job) with drift detection.
P2-9: Workflow registry is in-memory and demo-auth oriented#
- Evidence:
apps/isis/workflow-registry/src/services/workflow.service.ts:37(In-Memory Storage (replace with database))apps/isis/workflow-registry/src/app.ts:146(storageType: 'in-memory')apps/isis/workflow-registry/src/routes/workflows.ts:22andapps/isis/workflow-registry/src/routes/templates.ts:19(DEFAULT_OWNER = 'user_demo')
- Impact:
- Persistence and multi-tenant ownership controls are not production-ready.
- Remediation:
- Move workflows/versions/templates to Postgres.
- Enforce real auth context for ownership and access control.
P2-10: Output registry still uses demo owner default#
- Evidence:
apps/isis/output-registry/src/routes/outputs.ts:22(DEFAULT_OWNER = 'user_demo')
- Impact:
- Output ownership and access checks can be incorrect under multi-tenant operation.
- Remediation:
- Resolve owner from auth middleware/claims and enforce RBAC on output/provenance/lineage operations.
P2-11: Generation API readiness reports queue/storage as OK without backing checks#
- Evidence:
apps/isis/generation-api/src/app.ts:209(comment: checks would be added later)apps/isis/generation-api/src/app.ts:210(queue: 'ok')apps/isis/generation-api/src/app.ts:211(storage: 'ok')
- Impact:
- Operational readiness signals can be misleading.
- Remediation:
- Add real dependency checks (Redis queue probe, object storage probe) and fail readiness when unavailable.
P2-12: Model upload/download URL generation remains placeholder implementation#
- Evidence:
apps/isis/generation-api/src/services/model.service.ts:604-605(upload presign placeholder comment)apps/isis/generation-api/src/services/model.service.ts:636(download presign placeholder comment)
- Impact:
- Model management is not fully production-grade.
- Remediation:
- Integrate with real storage signing from
@oshun/storageand enforce content hash/size verification on completion.
- Integrate with real storage signing from
P2-13: Deployment config drift in Docker Compose Isis service paths#
- Evidence:
docker/docker-compose.yml:329(APP_PATH: apps/isis/api)docker/docker-compose.yml:359(APP_PATH: apps/isis/generator)- Actual directories are
apps/isis/generation-apiandapps/isis/gpu-worker.
- Impact:
- Local/container deployment can fail or run stale targets.
- Remediation:
- Correct compose app paths and add CI check validating compose
APP_PATHvalues against filesystem.
- Correct compose app paths and add CI check validating compose
P2-14: Test coverage does not validate real end-to-end execution and RunPod behavior#
- Evidence:
- E2E coverage is API-centric:
testing/e2e/tests/api/isis/jobs.e2e.ts,models.e2e.ts, etc. - Integration flow uses mocked event bus fixtures:
tests/integration/flows/isis-yemaya.integration.test.ts.
- E2E coverage is API-centric:
- Impact:
- Critical gaps (queue wiring, RunPod live execution, catalog compatibility) are not automatically detected.
- Remediation:
- Add contract/integration tests for:
- submit -> queue -> worker -> output registry end-to-end
- RunPod sandbox endpoint execution and status/cost accounting
- workflow catalog compatibility across loader, registry, executor
- Add contract/integration tests for:
Proposed Remediation Plan#
Phase 0 (0-2 weeks): Make execution real and deterministic#
- Wire generation API to BullMQ producer with consistent queue naming.
- Connect job lifecycle updates from worker callbacks.
- Integrate workflow catalog registration into executor startup.
- Fix schema mismatch between catalog and registry validator.
- Fix Docker Compose Isis app paths.
Exit criteria:
- A submitted job transitions to
completedwith non-placeholder output for at least one workflow in each major category (video,3d,avatar,fractal,utility).
Phase 1 (2-4 weeks): Productionize RunPod serverless operations#
- Implement real cost accounting and per-job history persistence.
- Add explicit capability mapping from Isis workflow IDs to RunPod handlers/endpoints.
- Add endpoint provisioning automation and drift checks.
- Add readiness and SLO metrics for queue/storage/provider health.
Exit criteria:
- RunPod jobs have auditable runtime seconds, calculated cost, and queryable historical records.
Phase 2 (3-6 weeks): Build Isis web app for full feature exposure#
- Create
apps/isis/webwith authenticated multi-tenant UI. - Expose all Isis surfaces (jobs, workflows, outputs/lineage, models, provider telemetry).
- Add category and workflow-ID based run experiences for all supported catalog entries.
Exit criteria:
- Every API capability in
@isis/clienthas UI coverage and route-level integration tests.
Phase 3 (ongoing): Hardening and confidence#
- Replace all simulated executor paths for production profiles.
- Add full E2E test pipeline in CI using ephemeral queue/db/storage and a RunPod test endpoint.
- Add release gates that block deploy on catalog-contract or provider-health regression.
Risk Register#
- High risk: Jobs accepted but not executed due to missing queue producer.
- High risk: Workflow contract mismatch causes silent feature non-availability.
- Medium risk: Cost/accountability blind spots for serverless usage.
- Medium risk: Missing web app blocks stakeholder-facing completeness objective.
Conclusion#
The Isis domain has substantial foundational assets (workflow corpus, provider adapters, service decomposition), but it does not yet meet the stated requirement of complete complex-workflow implementation, pay-per-second RunPod serverless readiness, and full Isis web app feature exposure. The highest-value next step is to close the submit-to-execution wiring and workflow schema integration before expanding UI and operational polish.