# Oshun slim ComfyUI worker (RunPod serverless)

The only worker image the Isis Chroma path uses. It is the stock
`runpod/worker-comfyui` base plus one node pack. **No model is baked into it.**

| Layer        | Pin                                                                                                         |
| ------------ | ----------------------------------------------------------------------------------------------------------- |
| Base image   | `runpod/worker-comfyui:5.10.0-base@sha256:e9d18d3db15839ebb10c10c824109fc2956f31ef7bf21916220c008855b0d538` |
| ComfyUI      | 0.34.0 (from the base image), CUDA 12.8, Python 3.12 in `/opt/venv`                                         |
| Node pack    | ComfyUI-RMBG `58f1947a11567a9f8b707223185570850e773856` (pyproject 3.1.0 + post-release fixes)              |
| Published to | `ghcr.io/greychimp/oshun/runpod-comfyui-slim` (private package)                                             |
| Built by     | `.github/workflows/runpod-comfyui-slim.yml` on GitHub runners — never on a laptop                           |

## What is in, what is out

> **A.01.09 (2026-09-14):** the `PozzettiAndrea/ComfyUI-SAM3` pack and
> ComfyUI-RMBG's vendored SAM 3 still node (`py/AILab_SAM3Segment.py`,
> `models/sam3/`) are removed from the image, with pycocotools, the `sam3` model
> folder, its symlink and the two SAM 3 manifest rows. Segmentation and video
> tracking run on ComfyUI core's SAM 3.1 nodes (`SAM3_Detect`,
> `SAM3_VideoTrack`, `SAM3_TrackToMask`) loading
> `checkpoints/sam3.1_multiplex_fp16.safetensors`. The smoke test fails the
> build if any removed class registers again. Passages below that describe
> `SAM3Segment` or ComfyUI-SAM3 are kept as history.

- **In:** ComfyUI core (Chroma nodes: `UNETLoader`, `CLIPLoader` type `chroma`,
  `T5TokenizerOptions`, `ModelSamplingAuraFlow`, `CFGGuider`,
  `BetaSamplingScheduler`, `SamplerCustomAdvanced`, inpaint/latent nodes) and
  ComfyUI-RMBG classes `RMBG`, `BiRefNetRMBG`, `SAM3Segment`.
- **In:** `pycocotools`, which is in neither the pack's `requirements.txt` nor
  ComfyUI's. `SAM3Segment` needs it: the SAM 3 source is vendored inside the
  pack (`models/sam3/`, not a PyPI package), and importing `sam3.model_builder`
  reaches a bare `import pycocotools.mask` through
  `sam3.model.sam3_video_inference` -> `sam3.model.sam3_video_base` ->
  `sam3.train.masks_ops`. Without it the node silently vanished from
  `/object_info` for a whole release (tracker C.12.18).
- **Out:** `groundingdino-py` (sdist needing a compiler the runtime base lacks).
  ComfyUI-RMBG's GroundingDINO-prompted `Segment`/`SegmentV2` nodes are
  therefore unavailable; text-prompted segmentation goes through `SAM3Segment`
  instead, which does not import it. The pack loads each module in its own
  `try/except`, so the missing optional dependency does not disable the rest —
  `AILab_SegmentV2.py` is the one entry on the smoke test's allowlist of node
  files that are permitted to fail loading.
- **Out:** any model file. See "Models" below.

## Contract with the generation API

Request (stock worker, `POST https://api.runpod.ai/v2/<endpoint>/run`):

```json
{
  "input": {
    "workflow": { "<nodeId>": { "class_type": "...", "inputs": {} } },
    "images": [{ "name": "input.png", "image": "data:image/png;base64,..." }]
  }
}
```

Response (`/status/<jobId>` once `COMPLETED`):

```json
{
  "output": {
    "images": [
      { "filename": "ComfyUI_00001_.png", "type": "base64", "data": "iVBOR..." }
    ],
    "errors": []
  },
  "executionTime": 34210,
  "delayTime": 91200
}
```

`executionTime` and `delayTime` (ms) are the billing truth; the Isis cost ledger
is computed from them, never from estimates. Keys the old Oshun handler
understood (`inputs`, `workflowMetadata`, `workflow_id`) are ignored by this
worker: the generation API renders the catalog JSON to a bare graph before
submitting (`ISIS_RUNPOD_WORKER_KIND=worker-comfyui`).

Worker environment variables that matter (set on the RunPod template):

| Variable                     | Purpose                                                             |
| ---------------------------- | ------------------------------------------------------------------- |
| `COMFY_POLLING_MAX_RETRIES`  | Handler polls ComfyUI this many times before giving up (set 1500)   |
| `COMFY_POLLING_INTERVAL_MS`  | Poll interval (set 250)                                             |
| `COMFY_LOG_LEVEL`            | ComfyUI verbosity (`INFO`)                                          |
| `BUCKET_ENDPOINT_URL` family | If set, outputs upload to S3 and `type` becomes `s3_url` (not used) |
| `WEBSOCKET`                  | Optional live-preview socket; not used by Isis                      |

## Models

Models are read from the network volume mounted at `/runpod-volume`, using
ComfyUI's folder layout under `/runpod-volume/models/`:

| Folder              | Contents                                                                                                                                      | Consumer                                          |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| `diffusion_models/` | Chroma1-HD, UnCanny v1.3 BF16, Photonic Fusion BF16                                                                                           | `UNETLoader`                                      |
| `text_encoders/`    | `t5xxl_fp16.safetensors`                                                                                                                      | `CLIPLoader` (type chroma)                        |
| `vae/`              | `ae.safetensors`                                                                                                                              | `VAELoader`                                       |
| `RMBG/BiRefNet/`    | `BiRefNet-portrait/-matting/-HR-matting.safetensors` + config files                                                                           | `BiRefNetRMBG`                                    |
| `loras/`            | Wan lightx2v + Animate relight; four Chroma LoRAs (C.12.01); LTX-2.3 distilled LoRA, IC-LoRA union control and motion-track (V.12.02/V.12.03) | `LoraLoaderModelOnly`, `LTXICLoRALoaderModelOnly` |

`extra_model_paths.yaml` in this directory maps every one of those folders;
`RMBG/` is additionally symlinked from `/comfyui/models/` at container start by
`entry.sh` (the image ships no dangling symlinks) because ComfyUI-RMBG
hard-codes `<models_dir>/RMBG` as its cache root. The authoritative inventory,
with sources, revisions and sha256, is
`infra/runpod/volumes/isis-chroma-models.manifest.json`; the loader that seeds
the volume is `scripts/isis/runpod-volume-loader.py`.

Rules:

- **Models → volume copy + manifest row. Custom nodes → image rebuild.**
- Pre-seed every model the catalog names. A worker that downloads at request
  time is spending billed GPU seconds on a network transfer.
- No quantized weights (GGUF / fp8 / int8 / DF11). The manifest lint rejects the
  filename patterns.

## Smoke test

`smoke-test.sh` runs inside the container on CPU (CI does this after every
build): boots ComfyUI, fetches `/object_info`, asserts every class the Isis
catalog needs is registered, fails on any custom-node import failure, and writes
`object_info.json` as a CI artifact. That artifact is what the generation API's
pre-submit class check is fed from.

**It is fatal to the build, and it has to stay that way.** It was briefly run
with `set +e` so that RunPod's console-only build log would not hide the reason
for a failure — and in that window the image shipped a release whose
`REQUIRED_CLASSES` listed `SAM3Segment` while the class was not registered at
all (C.12.18). A class list is only evidence if the check can fail the build.
The log problem is solved instead by dumping the entire smoke log to the build
output on failure and baking `build-manifest/smoke.log` + `smoke.status` into
the image, which `entry.sh` prints at container start where the worker log API
can read it.

Two failure modes it now catches that it did not before:

- ComfyUI-RMBG's `__init__.py` loads each node file inside a `try/except` that
  prints `Error loading <file>: <err>` and continues. ComfyUI does not count
  that as an import failure, so the class simply disappears. Every such line
  fails the smoke unless the file is on `ALLOWED_LOAD_FAILURES`. Two entries
  today, each a dependency this image drops on purpose: `AILab_SegmentV2.py`
  (`groundingdino-py`) and `AILab_Florence2.py` (`matplotlib`, which the pack's
  own `requirements.txt` does not list either; Florence 2 is not in the Isis
  catalog). The Florence 2 failure was found by this check on its first fatal
  build and had been happening silently in every image before it.
- The Dockerfile additionally performs `SAM3Segment`'s own module-level import
  (`from sam3.model_builder import build_sam3_image_model`) in its own layer, so
  a missing SAM 3 dependency fails with a real traceback rather than as an
  absent class.

## Video contract (tracker V.01.04)

**Video comes back in `images[]`.** Not a typo and not a bug to route around:
`SaveVideo` returns
`PreviewVideo.as_dict() = {"images": [...], "animated": (True,)}` (ComfyUI
0.34.0), and worker-comfyui's handler collects only `node_output["images"]`. So
mp4 / webm / mkv arrive in the same array as PNGs, as
`{filename, type: 'base64' | 's3_url', data}`. The adapter sniffs the bytes
rather than trusting the extension.

**Audio is always muxed, never a second artifact.** `SaveAudio` returns under an
`audio` key, which the handler does **not** collect — an audio-only save is
silently dropped. LTX-2.3's track therefore has to travel through
`CreateVideo(images, audio)` into `SaveVideo`. A graph that saves audio
separately produces nothing.

**S3 output, for clips too large to base64.** With `BUCKET_ENDPOINT_URL`,
`BUCKET_ACCESS_KEY_ID` and `BUCKET_SECRET_ACCESS_KEY` set on the endpoint, the
worker uploads the file and returns `type: 's3_url'` instead of the bytes. Those
three are attached to the endpoint from the environment by
`runpod-provision.mjs` and are listed in `desired-state.json` by **name only** —
a value there would be a committed credential.

**Video input arrives through `input/volume`.** `entry.sh` links
`/comfyui/input/volume` to `/runpod-volume/inputs`, so a clip PUT to the volume
at `inputs/<jobId>/<name>` is referenced by a `LoadVideo` node as
`volume/<jobId>/<name>` with no download-and-re-upload round trip. Small inputs
still ride `input.images[]`, which ComfyUI writes to disk as-is under the given
name (it does no content check, so a small mp4 travels that path fine).

### The packs, and why each one is here

ComfyUI 0.34.0 already ships **every** Wan, VACE, Animate and LTX-2.3 generator
node, so no pack is needed to make video. These six exist for what core does not
do:

| Pack                                    | Pin        | Why                                                                                                                                 |
| --------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `Kosinkadink/ComfyUI-VideoHelperSuite`  | `4d907bee` | Container I/O beyond core's `LoadVideo`/`SaveVideo` (`VHS_LoadVideo`, `VHS_VideoCombine`)                                           |
| `Fannovel16/comfyui_controlnet_aux`     | `59b1fc41` | DWPose and Depth Anything V2 preprocessing for control-guided edits                                                                 |
| `kijai/ComfyUI-WanAnimatePreprocess`    | `0e0b6a2a` | The pose/face detection Animate's driving video needs (`OnnxDetectionModelLoader`, `PoseAndFaceDetection`, `DrawViTPose`)           |
| `kijai/ComfyUI-GIMM-VFI`                | `4c9a3123` | Frame interpolation (`DownloadAndLoadGIMMVFIModel`, `GIMMVFI_interpolate`)                                                          |
| `lihaoyun6/ComfyUI-FlashVSR_Ultra_Fast` | `4820b3f0` | Video super-resolution (`FlashVSRNode`)                                                                                             |
| `PozzettiAndrea/ComfyUI-SAM3`           | `de0ff5d2` | SAM 3 **video** tracking (`SAM3VideoSegmentation`, `SAM3Propagate`, `SAM3VideoOutput`); ComfyUI-RMBG's `SAM3Segment` is stills only |

Three details that cost real time to establish and will cost it again if lost:

- **`ComfyUI-SAM3` cannot install itself here.** Its `install.py` and root
  `__init__.py` both import `comfy_env`, a pixi-based installer. At this SHA it
  ships no `requirements.txt` and no `prestartup_script.py`. The image deletes
  `install.py` and replaces the root `__init__.py` with a direct
  `from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS` — the
  pack's own `nodes/__init__.py` already assembles both. Its real dependencies
  were read out of `nodes/comfy-env.toml` into `requirements-video.txt`; the
  `[cuda]` extras (`cc_torch`, `torch_generic_nms`, `flash-attn`) are imported
  under `try`/`except` upstream and are deliberately absent, since `flash-attn`
  compiles and the runtime base has no compiler.
- **`FlashVSR-v1.1` is a folder name, not a label.** `nodes.py:140-141` joins
  `models_dir` with the model combo value verbatim, and calls
  `snapshot_download` of the whole repo when the directory is missing.
  `entry.sh` creates the symlink under exactly that name.
- **Never `opencv-python`.** Four packs request it; its GUI build links libGL,
  which a runtime image does not carry, so `import cv2` raises
  `libGL.so.1: cannot open shared object file`. `requirements-video.txt`
  installs `opencv-python-headless` once instead, and no pack's own requirements
  file is used directly.

## Registry credential and rotation

The package is private. RunPod pulls it with a Container Registry Auth made from
the GitHub username plus a personal access token with `read:packages` only
(env-master `GHCR_PULL_TOKEN`; registered by
`scripts/isis/runpod-provision.mjs registry-auth create`).

Rotation, in this order, so pulls never break:

1. Create the new token on GitHub; add it to the env file as `GHCR_PULL_TOKEN`.
2. `runpod-provision.mjs registry-auth create --yes` (new auth id), then
   `endpoint update --registry-auth <newId> --yes`.
3. `registry-auth delete <oldId> --yes`, then revoke the old token on GitHub.

## RunPod registry and endpoint landmines (2026-09-11)

- Images built by RunPod's git integration live in `registry.runpod.net` and can
  only be pulled by the endpoint/template that the build created. A plain
  template pointing at the same image name fails with
  `Failed to get Hub registry auth. Is this a template you have access to?`
- Never change a git-built endpoint through the GraphQL `saveEndpoint` mutation
  (used once to set GPU pools): the endpoint stopped placing workers afterwards
  even with a queued job. Use the console or the REST `PATCH /v1/endpoints/{id}`
  for in-place fields.
- "Active workers" (workersMin) triggers a RunPod cost warning e-mail every time
  it is raised; boot a worker with a queued job instead.
