V1 ComfyUI Workflows — SOTA Audit & Remediation (2026-07-03)

Scope: every ComfyUI workflow serving V1 (the 48-workflow canonical catalog in libs/isis/workflows/src/workflows/), the execution substrate that runs them (RunPod worker images, handler, on-demand model loader, endpoint manifests), the TypeScript builders/providers in libs/isis/ai-providers, and the split between hosted APIs (OpenRouter et al.) and self-hosted open-source models on RunPod. Method: exhaustive inventory → per-file adversarial audit (main line + 2 read agents) → web research of the July-2026 SOTA → remediation with tests → adversarial verification.

1. Verdict up front#

Before this audit the ComfyUI plane was a well-engineered shell around a generation core that could not run. The governance, dispatch, telemetry, and fail-closed seams (which earlier audits verified) are real and good — but the actual payloads were broken at every layer:

• The catalog renderer crashed on ~10 workflows (dict attribute-access in templates), produced dangling graphs for every conditional toggle-off in 8 more, submitted ComfyUI-rejected seeds (default -1) for all of them, and never returned video or audio outputs (only images was collected).
• The video family was architecturally impossible (SD1.5 motion modules paired with the SDXL checkpoint, plus AnimateDiff class names provided by no installed pack) and six of twelve worker images shipped with no handler at all (--target runtime-sdxl stopped before the stage that COPYs it).
• ~60 referenced custom node classes existed nowhere — not in ComfyUI core, not in the 9 baked community packs, and not in the in-repo pack (whose Lilith* classes were, in turn, never installed into any image).
• The adapter↔handler contract was broken end-to-end for arbitrary graphs: the handler executed the {nodes,…} wrapper verbatim, which ComfyUI rejects; only catalog-id jobs could run.
• The FLUX/SD3.5 TypeScript builders were dead code with per-family recipe errors, comfy-cloud's multi-provider layer was protocol-fabricated, and the unified pipeline contained code that forged vitest mock-call records to defeat its own load tests (removed; see §5).

All of the above is fixed, tested, and gated (see §4–§6). The honest scope boundary: fixes are verified by unit/contract tests and static gates in-repo; live GPU execution on RunPod remains a deploy-time activity (no RunPod credentials in this environment).

2. How the stack compares to the July-2026 SOTA#

Web-researched baseline (34 sources; key posts: BFL FLUX.2/klein, ComfyUI v0.25–0.27 changelogs, Wan/Hunyuan/LTX repos, OpenRouter image/video API announcements, worker-comfyui 5.8.6):

Overall: after remediation the ComfyUI plane is correct, coherent, and mid-tier by deliberate choice (SD1.5/SDXL fit the baked-adapter ecosystem and the contemplative product's aesthetic), with the frontier tier served via hosted APIs (Stability, OpenRouter FLUX.2 Pro, fal LTX). The open-frontier upgrades (FLUX.2-klein endpoint, LTX-2 video endpoint, TRELLIS.2 3D, SeedVR2) are now pure endpoint additions on a substrate whose gates verify them.

3. OpenRouter + RunPod wiring assessment#

• LLM traffic: already expertly wired — @shared/ai routes OpenRouter-first (model-pattern table, fallback order), gated on OPENROUTER_API_KEY, fail-closed. Unchanged.
• Hosted media: was direct-to-vendor only (Stability/fal/Suno/ElevenLabs). OpenRouter's 2026 media APIs were entirely unused. Fixed: the customer image path can now route through OpenRouter's unified Image API behind the same key/billing as LLM traffic, selected by OSHUN_IMAGE_PROVIDER, with no silent cross-vendor fallback and the same fail-closed release-gate measurement path (data-URI outputs flow through the real moderation scan).
• Open-source models on RunPod: the dispatch machinery (worker app → @oshun/gpu-dispatcher → api.runpod.ai/v2; per-endpoint desired-state, warm/cold policy, canary routing, circuit breaker) is real. The payload layer beneath it was broken (§1) and is now fixed + gated: the manifest generator validates every workflow's node classes against what images actually install and every model against baked ∪ on-demand-resolvable ∪ gated, failing CI on any drift.

4. Defects found and fixed (execution layer)#

Renderer/handler (docker/runpod/comfyui — mirrored 1:1 by the new TS renderer):

1. Dict dot-access templates (configs[key].prop) crashed Python eval — AttrDict contexts; both engines now accept both styles.
2. _condition node drops left dangling links — downstream rewiring with explicit _condition_passthrough, payload-name heuristics, transitive chains, and source-chain consumer-input deletion.
3. Catalog default seed -1 failed ComfyUI validation — negative seeds normalize to random; KSamplerAdvanced/SamplerCustom/RandomNoise seeds land on noise_seed (previously written to an ignored key = silent non-determinism).
4. Video (gifs/videos) and audio (audio) history outputs were never collected — all media kinds now returned (back-compat images preserved); S3 content types per artifact.
5. {nodes,metadata} wrapper executed verbatim (ComfyUI rejects) — handler unwraps to the bare node map; flat input overrides map onto node inputs instead of crashing.
6. _sole expression regex bug: multi-expression strings starting AND ending with {{…}} were evaluated as one expression (latent in the original handler; exposed and fixed).

Workflow catalog (all 48 render clean through every boolean-toggle combination in BOTH engines — test_catalog_renderer.py 18 tests, catalog-renderer.spec.ts 15 tests, plus the pre-existing 89-test schema suite): 7. Video family → SD1.5 + AnimateDiff-Evolved Gen1 (ADE_* classes, v3 motion module, uniform context options, multival motion scale) + VHS; SD1.5 resolutions; frame caps; fps defaults at AD-native 8. 8. consistent-character → real FaceID Plus v2 chain (unified loader + lora + insightface + ViT-H); style-transfer → all-SD1.5 stack (baked lineart ControlNet + IPAdapter PLUS) with honest _condition custom-image path (was: fake enabled inputs silently ignored by ComfyUI). 9. 3D → core ImageOnlyCheckpointLoader + StableZero123_Conditioning_Batched (was: fabricated Zero123Plus custom chain); grid/caption nodes consume the descriptive inputs for real. 10. Audio → ComfyUI-native Stable Audio Open & ACE-Step graphs + isis DSP (loudness normalize/limiter/de-esser) + real Kokoro TTS node; fabricated emotion input removed rather than faked. 11. Authoring bugs: fractal-morph none lookup, guided-journey undefined journey_prompt_schedule, image-to-video {{node_output[4]}} (graph-output-in-template) → BLIP caption wired by node link, RepeatLatentBatch.latent → samples, mandala-3d string-fed contrast, dead reference_image input, duration_seconds min>default conflicts. 12. custom_nodes_required / models_required regenerated from actual graphs.

Node pack (libs/isis/comfyui-nodes): every remaining custom class implemented with real algorithms (fractal escape-time renderers, Reinhard/Lab color transfer, BS.1770 loudness, lookahead limiter, zodiac/constellation vector overlays, print pipeline incl. ICC + 16-bit TIFF, temporal EMA processors, BLIP captioning, Kokoro ONNX TTS, …) and the pack is now actually installed into the worker images. See tests/test_catalog_support_nodes.py.

Infra: 13. Dockerfile: parameterized final stage (every flavor ships the handler — previously sdxl-target images had none), pinned ComfyUI, strict per-pack clones with SHA build manifest, dead runwayml SD1.5 URL → Comfy-Org archive, AnimateDiff-Evolved/ReActor/isis packs added, identity + motion + audio model bakes, new model-dir symlinks. 14. On-demand loader: KNOWN_MODELS matched by filename (was: key-only, so the HF fallback never fired for workflow-extracted specs); catalog extended with every model the catalog needs; model-type dirs for motion/clip_vision/ ipadapter/insightface/tts. 15. Manifest generator: derives classes from graphs, validates against provided classes and resolvable models, syncs the docker catalog + isis pack from canonical sources, exits non-zero on violations (the previous generator trusted hand-written fields and emitted provably wrong inventories, e.g. audio endpoints with requiredCustomNodeClasses: []). 16. Build config: ComfyUI targets → final + MODEL_FLAVOR; video endpoints → sd15 flavor; diffusers sd/flux endpoints no longer claim ComfyUI catalog workflows they cannot execute.

5. Defects found and fixed (TypeScript providers)#

From the adversarial audit of libs/isis/ai-providers (all fixed with tests):

• unified-generation-pipeline: removed recordSyntheticProviderCall (production code that pushed forged entries into vitest mock.calls so load tests passed without real provider calls); removed wrong-result cross-request dedup of stochastic generations; placeholder success string → loud provider-contract error; metrics.cost type corruption fixed; routing now updates live observed latency/success instead of scoring identical hardcoded capabilities; quality-labeled integrity score renamed honestly; real request timeouts; non-recoverable codes fail fast.
• runpod-adapter: real static validateWorkflow (was hardcoded {valid:true} ignoring its argument); true semaphore for maxConcurrentJobs (was a time-refilling token bucket that drifted above the cap); analytics double-count/inflation fixes; estimates no longer presented as billed usage; submits the bare node map natively.
• flux-workflows: serialize emits only the node map; core-only inpaint recipe (SetLatentNoiseMask); EmptySD3LatentImage; ModelSamplingFlux; real ControlNet inventory (Shakker Union Pro 2.0 + union-type node, with the required vae input); schnell guidance rejected instead of silently ignored; 64-bit-safe seeds; wiring spec added.
• sd35-workflows: rewritten to true API format (was a bespoke structure no converter consumed); TripleCLIPLoader (the distributed checkpoint has no text encoders); ModelSamplingSD3 shift 3.0; sgm_uniform defaults; turbo profile (4 steps / cfg 1); ControlNetApplySD3 with vae; valid blur/scale params; flow-incompatible karras/exponential style presets replaced; wiring spec added.
• lora-manager: cache cleanup actually deletes files and reports real freed bytes; LRU eviction enforces the configured cap; spliceLoraChain performs the documented rewiring instead of returning orphan nodes.
• comfy-cloud: honest scope reduction — the real self-hosted ComfyUI protocol fixed end-to-end (correct /history/{id} parsing, /view params, provider job-id tracking for cancel, non-zero cost), RunPod path speaks the worker envelope, and the protocol-fabricated providers (modal/vast/ comfy-org) now fail loud as unsupported_provider instead of pretending.

6. Gates that keep it fixed#

• docker/runpod/comfyui/tests/test_catalog_renderer.py — Python renderer contract over the real catalog (defaults + every boolean toggle + seeds).
• libs/isis/workflows/src/templates/catalog-renderer.spec.ts — the TS twin with the same assertions (cross-engine drift is a test failure).
• scripts/isis/generate_runpod_endpoint_image_manifests.mjs — class/model supply-chain validation; non-zero exit on any gap; syncs docker contexts.
• Existing schema/lint suites (89 tests) extended by the _condition_passthrough schema key and duration-bound fixes.
• Provider specs in libs/isis/ai-providers and the BFF OpenRouter provider tests (fail-closed selection semantics included).

7. Honest remaining boundaries#

• No live GPU run in this environment: the contract tests prove graph validity/renderability and wire shapes, not pixel output. First deploy should run one catalog workflow per endpoint (the handler's preflight + inventory ops make this a single workflow_id smoke call each).
• Gated weights (stable_zero123.ckpt, stable_audio_open_1.0) resolve on demand only with HF_TOKEN on the endpoint — fail-closed otherwise.
• Frontier-tier open models (FLUX.2-klein, LTX-2, TRELLIS.2, SeedVR2) are endpoint additions, deliberately not bolted onto the existing images: each adds 5–60GB of weights and its own GPU class. The substrate (flavored images, KNOWN_MODELS, validation gates, catalog schema) now supports adding them without repeating this audit's failure modes.
• add_detail.safetensors bakes from the OedoSoldier HF mirror; if that mirror moves, the on-demand entry is the single place to update.

Addendum (same day): granular 2026-SOTA pass over the catalog#

A second remediation wave took the workflows themselves from "correct but 2023/24-era craft" to expert 2026 implementations. All node contracts and artifact names below were wire-verified against ComfyUI core sources, official workflow templates, and HuggingFace trees before authoring.

Within-tier craft (every existing workflow):

• 26 txt2img stills: two-pass hires-fix (1.5x bislerp latent upscale + 14-step dpmpp_2m/karras denoise-0.45 refine) and FreeU_V2 with family-tuned parameters, both behind default-on booleans riding the conditional-rewire contract.
• Curated finetune checkpoints via enum inputs: SDXL stills default to Juggernaut-XL v9 (RealVis V5 / base as options); SD1.5 identity/video workflows default to DreamShaper 8 — all ungated direct-download HF hosting, in KNOWN_MODELS, baked per flavor.
• 4 portrait workflows: Impact FaceDetailer detect-and-inpaint pass (exact wire signature incl. cycle/wildcard/SAM-optional fallback, Ultralytics bbox chain, face_yolov8m) behind face_detail.
• All 7 AnimateDiff workflows: v3 domain-adapter LoRA (0.7, model-only) behind motion_adapter; all 9 video workflows: RIFE 4.9 2x interpolation behind smooth_motion (VHS fps doubles via int()-coerced expression — the TS evaluator caught enum-string fps that Python eval would have silently string-multiplied) and optional per-frame 2x RealESRGAN enhancement behind frame_upscale.
• 2023 prompt boilerplate (masterpiece/8k idiom) stripped from 32 templates; domain prompt libraries kept.

New flagship tier (2026-generation open models, all fleet-sized):

• flagship/z-image-sacred-art + flagship/z-image-portrait — Z-Image-Turbo (6B S3-DiT, Apache 2.0): UNETLoader + CLIPLoader [lumina2 → Qwen3-4B] + ModelSamplingAuraFlow shift 3, 8-step res_multistep/simple at cfg 1.0, prose prompting.
• flagship/wan-video — Wan 2.2 TI2V-5B (Apache 2.0): text-to-video and image-to-video in one graph (the conditional start-image LoadImage cleanly deletes the optional latent input in t2v mode), uni_pc/simple, 20 steps, cfg 5, shift 8, 1280x704 up to 121 frames @24fps, core CreateVideo/SaveVideo mp4.
• 3d/hunyuan3d-asset — ComfyUI-native Hunyuan3D 2.1 image-to-mesh (GLB) with an honest three_d_package contract (this graph produces the mesh; texturing is downstream).
• Audio upgraded in place: music → ACE-Step 1.5 turbo (new TextEncodeAceStepAudio1.5 node family; bpm/key/time-signature control), SFX → Stable Audio 3 Small SFX (May 2026 weights) + T5Gemma encoder.
• New endpoints comfyui-flagship-image / comfyui-flagship-video (L40S/4090-first pools) with dedicated Docker flavors baking the weights; three-d bakes Hunyuan3D 2.1. SeedVR2, Impact-Subpack and Frame-Interpolation packs installed with SHA build manifests.

Verification: 52-workflow catalog green on all four gates (Python render contract, TS render contract + 175-test suite, validate CLI, supply-chain generator at 0 violations across 14 endpoints).

Addendum boundaries: Stable Audio 3 small reference settings are inferred from the medium-distilled template (documented in-graph); ACE keyscale is a fail-loud free string ('E minor' template-confirmed); SeedVR2 is installed and supply-chain-registered but intentionally not yet wired into a catalog workflow (its device enum is worker-topology dependent — 'cuda:0' single-GPU default verified, operator graphs can use it today). First deploy should smoke one workflow_id per endpoint, now including the two flagship endpoints.

Addendum 2 (same day): control-surface pass — supply-chain URL audit, composition/controllability family, dual-path completion#

A third pass focused on what the first two deferred: the control surface (composition, region editing, structural control, series consistency), the deferred SeedVR2 wiring, the OpenRouter Video API, and a live URL audit of the model supply chain. Node/API contracts were wire-verified against primary sources (ComfyUI v0.27.0 tag, pack sources on GitHub, HF tree APIs, OpenRouter live API) before authoring.

Supply-chain defects found by live URL sweep (all fixed)#

Every Dockerfile bake URL and every KNOWN_MODELS source was HEAD-checked:

1. All 8 SD1.5 ControlNet .safetensors URLs were dead (404) — lllyasviel/ControlNet-v1-1 hosts .pth only. Because bakes soft-fail (|| echo WARN), every worker image shipped with zero ControlNet models and the on-demand fallback 404'd too: style-transfer and animatediff-controlnet were broken on deploy. Fixed to the maintained comfyanonymous/ControlNet-v1-1_fp16_safetensors packaging with _fp16 filenames end-to-end (Dockerfile, KNOWN_MODELS, both workflows, renderer test fixture); all 8 URLs verified 200.
2. normal_map control mode had no resolvable model at all — control_v11p_sd15_normalbae was absent from KNOWN_MODELS; the supply-chain gate missed it because enum-selected models were invisible to validation (it skipped every templated model input). The generator now sweeps enum options through the two catalog template forms ({{table[input].field}}suffix, {{input}}suffix) and validates every option's model; templated model slots it cannot enumerate now require a hand-declared models_required (mirrors the templated-class rule). The TS render spec also gained the enum-option sweep the Python suite already had (renders every declared option, not just defaults).
3. inswapper_128.onnx bake got 401 — the Gourieff/ReActor model repo went auth-gated; switched to the public ReActor dataset mirror (verified 200). The KNOWN_MODELS entry uses the direct URL because the hf:// scheme has no dataset repo-type support.
4. clip-vision-vith on-demand entry could never produce the filename the IPAdapter loader globs for (hf:// keeps the source basename model.safetensors) — switched to the direct-URL downloader which writes to the entry's canonical name.
5. sd21 (stabilityai SD 2.1) went HF-gated — documented as gated; stable-audio-open 401 is the already-declared gated path (unchanged).
6. Latent type errors in @isis/workflows surfaced and fixed (bare tsc --noEmit checks nothing there — empty root include; the lib check is -p tsconfig.lib.json): 'L40S' missing from the registry gpuRequirement union (4 flagship entries), and an unknown-typed class_type assignment in the TS renderer.

New composition family (SDXL tier) — the controllability gap closed#

The catalog had strong what controls (prompt libraries, finetune enums, identity) but almost no where/how controls: ControlNet was used by only 2 of 52 workflows, and there was no inpaint, no outpaint, no regional prompting, no SDXL style reference. Six new workflows (58 total), all rendering green through every enum option and boolean toggle in BOTH engines, all models resolvable at 0 violations:

• composition/controlled-composition — xinsir Union ControlNet SDXL (promax) baked on the sdxl flavor + core SetUnionControlNetType (exact grouped enum strings from control_types.py); depth (DepthAnythingV2), pose (DWPose), edge (Canny), lineart preprocessors selected by config table; preprocess=false feeds a ready control map via the conditional-rewire contract; strength + schedule-end controls; hires-fix/FreeU house conventions.
• composition/regional-composition — core percentage-area conditioning (ConditioningSetAreaPercentage + ConditioningCombine): base prompt plus up to two independently prompted, placed, strength-weighted regions.
• composition/style-reference-set — IPAdapter PLUS (high strength) on SDXL (ip-adapter-plus_sdxl_vit-h + ViT-H encoder, both baked and on-demand-resolvable): a reference image style-locks a batched variation set; weight_type exposes 'style transfer' / 'prompt is more important'.
• composition/inpaint-edit — core-only differential inpaint for standard checkpoints: LoadImage alpha mask → GrowMask → VAEEncode + SetLatentNoiseMask + DifferentialDiffusion (VAEEncodeForInpaint erases masked content and is only correct for dedicated inpaint models — chain chosen per core source).
• composition/outpaint-expand — core ImagePadForOutpaint (direction table × expand_pixels arithmetic, feathered mask) + differential fill at denoise 1.0.
• video/video-restore — SeedVR2 v2.5 wired (previous deferral closed): exact SeedVR2LoadDiTModel/SeedVR2LoadVAEModel/ SeedVR2VideoUpscaler signatures from pack source; 3B-fp8/3B/7B/7B-sharp DiT enum (registry filenames verified; weights auto-download with SHA-256 into models/SEEDVR2 — declared packManagedModels); 4n+1 temporal batching enforced by enum; six color-correction modes; seed normalization extended to the class in both renderers (32-bit random fits its 2^32-1 cap).

New core classes admitted deliberately: SetUnionControlNetType, DifferentialDiffusion, ImagePadForOutpaint, ConditioningSetAreaPercentage. New category composition rides the comfyui-sdxl endpoint; taxonomy registry now maps 58/58 with control modalities and consistency dimensions per workflow.

OpenRouter Video API (dual-path completion)#

OSHUN_VIDEO_PROVIDER=openrouter now routes the customer video path through OpenRouter's async Video API (wire shape live-verified: 202 + polling_url → poll to completed → unsigned_urls are auth-required, so the adapter downloads the MP4 with the same key and emits a self-contained data: URI — never a URL a client cannot fetch). Default model alibaba/wan-2.7 (override via OSHUN_OPENROUTER_VIDEO_MODEL; catalog live-verified: Veo 3.1, Seedance 2.0, Sora 2 Pro, Kling 3.0, Hailuo 2.3…). Fail-closed per selected vendor, no silent fallback; the real ffprobe/ffmpeg + frame-safety measurement path now materializes data: URIs to temp files (argv limits make direct data:-URI probing infeasible); the executor attaches the real provider name instead of the hardcoded 'ltx' literal (M19 pattern).

Honest boundaries after this pass#

• Hunyuan3D 2.1 texture stage stays deferred deliberately: ComfyUI core v0.27.0 has shape-stage nodes only; the community paint pipeline (visualbruno/ComfyUI-Hunyuan3d-2-1) requires two compiled C++ extensions and ~21 GB VRAM for texturing — a dedicated-endpoint decision, not a bolt-on. The shape→GLB path remains real and gated.
• Contract tests prove renderability, wire shape, and supply-chain resolvability — not pixels. First deploy: smoke one workflow_id per endpoint (unchanged advice), now including composition/* on comfyui-sdxl and video-restore on cinematic-video.
• SeedVR2 7B tiers want ≥48 GB VRAM at 1080p; the L40S pool fits 3B-fp8 comfortably — the enum default is the fast 3B build for that reason.