# Runbook: Isis Chroma on RunPod — local dev stack

Tracker: `ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md` (C.10). Evidence:
`docs/agents/isis-chroma-runpod-evidence.md`. Design: ADR-0005.

The generation API runs on this machine; the GPU work runs on the RunPod
serverless endpoint (max one worker) against the network volume. Nothing here
needs a local GPU.

## 0. Machine rules (CLAUDE.md, binding)

- `memory_pressure | tail -1` before every heavy start; below 35 % free → tear
  down first.
- One Vite/tsx server per session. Track PIDs in `/tmp/claude-<session>.pids`.
- Servers die with the session:
  `lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(4010|5180)'` must be empty at the
  end.

## 1. Infra

```bash
docker compose -f docker/docker-compose.dev.yml up -d postgres redis
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(5432|6379)\b'   # both must listen
```

Postgres is `postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev`. The jobs
table is created on first boot (`CREATE TABLE IF NOT EXISTS jobs`). Redis backs
the daily USD ledger. MinIO is not needed until C.07 lands (outputs travel as
data URLs inside the job record until then).

## 2. Env recipe (throwaway, never committed)

Keys come from `~/Desktop/workspace/drawer/env-master.env`; copy by name, verify
by length only, delete the file at session end.

```bash
S=/tmp/isis-chroma-env.sh
{
  grep -E '^(RUNPOD_API_KEY|RUNPOD_S3_ACCESS_KEY|RUNPOD_S3_SECRET_KEY|CIVITAI_API_KEY|GHCR_PULL_TOKEN)=' \
    ~/Desktop/workspace/drawer/env-master.env | sed 's/^/export /'
  echo "export JWT_SECRET=$(openssl rand -hex 32)"   # throwaway, never in env-master; see §2b
  cat <<'EOF'
export NODE_ENV=development
export PORT=4010
export BASE_URL=http://127.0.0.1:4010
export DATABASE_URL=postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev
export REDIS_URL=redis://127.0.0.1:6379
export ISIS_QUEUE_PRODUCER_ENABLED=false        # no BullMQ consumer exists locally
export ISIS_INLINE_JOB_RUNNER=true              # the API runs its own jobs (C.06.06)
export COMFYUI_PROVIDER=runpod
export ISIS_RUNPOD_WORKER_KIND=worker-comfyui   # API renders catalog → bare graph
export RUNPOD_COMFYUI_ENDPOINT_ID=6jye7vngg4i685   # rebuilt 2026-09-12 (V.03.04 note); see §2g
export RUNPOD_REGION=EU-RO-1                    # provenance ledger `region` (the volume's datacenter)
# export ISIS_PROVENANCE_ALLOW_UNWATERMARKED_IMAGES=true  # C.12.13 narrowed this
#   and OFF is now the right default. Images are watermarked for real; this flag
#   only decides what happens to the occasional output the stage could not mark
#   (off = that job's result is refused at completion, the GPU time having
#   already been spent). It no longer gates submission.
export ISIS_COST_RATE_RUNPOD_USD_PER_SECOND=0.000439   # RTX 5090 flex ($1.58/h) — the priciest
#   card the rebuilt endpoint accepts, NOT the 4090's 0.000306. A gpuIds list means "any of
#   these" and the ledger cannot know which card ran, so it has to bill the dearest one:
#   under-billing is the direction that stops the budget firing. The preflight fails on this.
export ISIS_RUNPOD_DAILY_BUDGET_USD=3
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=true
export ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD=10
export ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD=25
export ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW=3
export ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS=600
export ISIS_RUNPOD_BUDGET_FAIL_CLOSED=true
export ISIS_COST_RATE_DEFAULT_USD_PER_SECOND=0.000306
# export ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS=3600000  # C.12.17; default 3600000 (1 hour)
#   The UNBILLED queue + cold-boot allowance, held separately from the GPU
#   execution cap so a slow image pull cannot get a rendering job cancelled and
#   re-billed. **Raised from 10 min to 1 HOUR on 2026-09-12.** Ten minutes was
#   set when we thought a cold start's delayTime was billed, so waiting looked
#   expensive; V.11.01 measured that it is not (ledger $1.20, account $0.29).
#   Waiting is free, and EU-RO-1 needed FOUR attempts to place one clip — twice
#   a card arrived within a minute of a shorter deadline. Lower it (e.g. 45000)
#   for the kill-switch drills C.11.12 / V.11.04, which measure exactly this
#   wait; restore afterwards. A zero or malformed value is ignored, not obeyed.
export VITE_ISIS_GENERATION_API_URL=http://127.0.0.1:4010
export CORS_ORIGINS=http://127.0.0.1:5180   # the API rejects the dashboard's XHR without it
# export ISIS_RUNPOD_BUDGET_ALARM_PROJECT_ID=<project id>   # C.04.06: project whose webhook
#   subscriptions receive runpod.budget.warning / .exhausted. Unset = alarms log only.
#
# --- Video extension (V.00.02). Commented out until V.03.04 creates the second
# --- endpoint: an unset id is a loud failure, an empty one is a silent
# --- submission to nowhere. Uncomment the block as a whole, id first.
export RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID=2lezocymx2rgot   # V.03.04, created 2026-09-12; `motion` workflows route here
export ISIS_COST_RATE_RUNPOD_VIDEO_USD_PER_SECOND=0.001647   # H200 141 GB flex ($5.93/h) — the
#   dearest card in the EU endpoint's list (user decision 2026-09-13; 0.002400
#   was the deleted US endpoint's B200). A gpuIds list means "any of these" and
#   the ledger cannot know which card ran, so it bills the dearest; the card that
#   usually runs (RTX PRO 6000 Blackwell) settled at $0.00105/s on 2026-09-13.
# --- Model cache (option B, L.04). Leave ISIS_RUNPOD_CACHE_VOLUME_ID unset until L.06.01 creates the
# --- 400 GB cache volume: unset, cache gating is off and jobs run against the old full volume.
# export ISIS_RUNPOD_CACHE_VOLUME_ID=<isis-model-cache volume id>
export ISIS_RUNPOD_CACHE_SYNC_MB_PER_SECOND=40     # MiB/s; L.06.02 median with three transfers sharing Hetzner (about 75 alone; the L.00.01 probe read 20.8)
# export ISIS_RUNPOD_CACHE_AUTOSYNC=true            # hold jobs as awaiting_cache instead of 409
# --- Hetzner storage guard (L.07.02): outputs retention + the 900 GB account alarm. Off unless enabled,
# --- and the sweep is a dry run until the second line says false. Needs the HETZNER_APP_DEV_* names.
# export ISIS_HETZNER_STORAGE_GUARD_ENABLED=true
# export ISIS_HETZNER_OUTPUTS_RETENTION_DRY_RUN=false   # only after a dry-run log line looks right
# export ISIS_HETZNER_STORAGE_ALARM_PROJECT_ID=<ops project>   # falls back to ISIS_RUNPOD_BUDGET_ALARM_PROJECT_ID
export ISIS_WORKER_OUTPUT_FETCH_HOSTS=hel1.your-objectstorage.com   # C.12.06: hosts the API may
#   fetch presigned worker outputs from (the bucket the worker's uploader writes
#   to). Unset = only the RunPod volume's own S3 URL is accepted.
# export ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6      # second ledger key, separate from the image one
# export ISIS_RUNPOD_VIDEO_INPUT_INLINE_MAX_BYTES=  # V.05.01 measures it; above this, inputs go to the volume over S3
# export RUNPOD_VOLUME_ID=5bst2pgw7y                # the model cache volume, the only one since 2026-09-13
# export RUNPOD_S3_ENDPOINT=https://s3api-eu-ro-1.runpod.io   # SigV4 host for that bucket
EOF
} > "$S"; chmod 600 "$S"; source "$S"
echo "RUNPOD_API_KEY len=${#RUNPOD_API_KEY} S3 len=${#RUNPOD_S3_ACCESS_KEY}/${#RUNPOD_S3_SECRET_KEY} civitai len=${#CIVITAI_API_KEY} ghcr len=${#GHCR_PULL_TOKEN} jwt len=${#JWT_SECRET} endpoint len=${#RUNPOD_COMFYUI_ENDPOINT_ID} video-endpoint len=${#RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID}"
```

## 2c. What the first live run (2026-09-11) made mandatory

- **Consent id on every job.** The provenance ledger (active whenever
  `DATABASE_URL` is set) refuses to complete a generation without `consentId`;
  the API now refuses such jobs at submit, before billing. The Studio's Chroma
  presets attach `consentId: consent_operator_isis_dev_studio` (operator's own
  consent for a typed prompt); curl submissions must pass `parameters.consentId`
  themselves.
- **Images are watermarked (C.12.13, was: unwatermarked by policy).** The
  worker-comfyui path now embeds a provenance payload —
  `isis:v1;job=…;consent=…;wf=…;model=…;produced=…` — in the blue channel's low
  bit of every inline PNG output, before the bytes are persisted, and records
  `sha256(payload)` as the bundle's `watermarkHash`. The payload is framed with
  a CRC and repeated (512 times over a 1024² image), so damage to part of the
  picture still yields it back. Measured on the C.11 fixture: **+1.9% file size
  versus ComfyUI's own encoder, +0.5 s, every sample moved by at most 1**.

  What it does not do, on purpose: spatial LSB survives a lossless re-save and
  the storage path, but **not** JPEG/WebP re-compression, resizing, cropping or
  rotation. It is a provenance marker for the artifact Isis stores and serves,
  not a forensic watermark against a determined adversary; a transform-domain
  (DWT-DCT) version is separate, larger work.

  `ISIS_PROVENANCE_ALLOW_UNWATERMARKED_IMAGES` survives with a narrower job and
  should be **off**: it decides only what happens to an output the stage could
  not mark. Off refuses that job's result at completion; on keeps it with
  `watermark: { applied: false, reason }` recorded. A job with no still image to
  mark (a clip, a mask-only flow) is not covered by it either way.

- **Runtime, settled 2026-09-11 21:39Z.** The endpoint is `6l1f3tku7z2zoy`
  (isis-comfyui-chroma), built by RunPod's git integration from branch
  `runpod-comfyui-slim`. Its active release `d677f914` (commit `105ad3cf`)
  **works** — it creates containers, boots ComfyUI and renders. The interim
  stock-image endpoint `ac988l8is5axkf` that produced the first image was
  deleted and is no longer needed; template `zxojew9tyo` survives as a fallback
  only.
- **If the endpoint stops placing workers, re-save it in the console.** After a
  GraphQL `saveEndpoint` call this endpoint went dead: zero workers in every
  state with jobs queued, and a REST `PATCH {"workersMax":1}` did not revive it.
  **Manage → Edit endpoint → Save Endpoint**, re-applying the same
  configuration, fixed it immediately. Never call GraphQL `saveEndpoint` on a
  git endpoint; if one is already wedged, the console form is the cure, not a
  rebuild.
- **"Low Supply" is not a blocker.** The edit dialog labels EU-RO-1's selected
  pools Low Supply / Unavailable, yet a worker was placed within ~40 s. Do not
  conclude a capacity problem from that label alone, and do not propose
  migrating datacenters over it — a network volume cannot move, so migrating
  costs a second 500 GB volume plus re-seeding ~70 GB.
- **Cheapest possible liveness probe** (costs nothing if no worker boots, ~$0.03
  if one does): submit a model-free graph and purge afterwards.

  ```bash
  curl -s -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
    -d '{"input":{"workflow":{
      "3":{"class_type":"EmptyImage","inputs":{"width":64,"height":64,"batch_size":1,"color":8421504}},
      "4":{"class_type":"SaveImage","inputs":{"images":["3",0],"filename_prefix":"smoke"}}}}}' \
    "https://api.runpod.ai/v2/$RUNPOD_COMFYUI_ENDPOINT_ID/run"
  # then: POST https://api.runpod.ai/v2/<ep>/purge-queue
  ```

- **GPU pools, not GPU names.** EU-RO-1 (the volume's datacenter) had no free
  4090 and does not offer A5000/3090; both endpoints use pools
  `ADA_24,AMPERE_24,BLACKWELL_32` set through GraphQL `saveEndpoint` (the REST
  PATCH silently ignores `gpuTypeIds`). Renders land on RTX PRO 4500 Blackwell.
- **Measured cost.** Cold run: 53 s queue/boot + 150 s GPU = $0.083 at the 4090
  rate; the model load from the volume (T5 ~40 s, Chroma1-HD ~65 s) is most of
  the GPU time. Ledger rate stays 0.000306 $/s until RunPod's invoice for the
  Blackwell pool is compared (C.11.13).

### Restarting the API mid-job is survivable now (C.12.16)

A job that was in flight when the API process died used to be lost outright: the
inline runner claims only `pending`/`queued`, so the row sat at `running`
forever while RunPod finished the render and billed for it (job `7d4280f9`,
148.98 s, gone). Two things changed:

- **The RunPod job id is written to `job.metadata` the instant RunPod accepts
  the submission**, before the first status poll — `providerJobId`,
  `providerEndpointId`, `providerDispatchedAt`.
- **The runner sweeps once at startup**, before it claims anything. A `running`
  job whose owning process is provably gone (`inline-<pid>` and the pid is dead,
  or the pinned `ISIS_INLINE_JOB_RUNNER_ID` is this process's own) is put back
  in the queue with its dispatch record intact. The next claim **re-attaches**
  to the existing RunPod job rather than submitting a new one, so nothing is
  rendered or billed twice, and the cost ledger is idempotent per job id anyway.

What it does NOT do, on purpose: a job whose worker id cannot be proved dead is
left exactly where it is, and logged as `left` with the reason. Two processes
running the same job would double-bill it, which is worse than a stuck row. A
job that has no `providerJobId` (dispatched before this change, or the write
failed) is failed with `ORPHANED_WITHOUT_PROVIDER_JOB_ID` rather than left lying
about being `running` — nothing can say which render was its.

Watch the startup log for `Re-queued an orphaned job for re-attachment` and
`Re-attaching to an already-dispatched RunPod job instead of submitting`. Note
the CLAUDE.md rule still stands for its own reasons: do not edit app source
while a run is in flight, because `tsx watch` recompiles under it.

### 2e. Cold boot on this endpoint got much slower when the video packs landed

Measured 2026-09-11, before the video packs: a fresh EU-RO-1 host pulled the
14.7 GB worker image in **~4 minutes** (job `d1e87d06`).

Measured 2026-09-12, after V.01.01 added six video node packs and their
dependencies (cupy, triton, pycocotools, timm, the controlnet_aux annotators): a
fresh host was still `initializing` **after 15 minutes**, and
`runpod-catalog-run.mjs` gave up in `IN_QUEUE` without the job ever starting.
Nothing was billed — `delayTime` is not GPU time — but nothing ran either.

What this means in practice:

- **A cold start here is minutes, plural, and possibly many.** Plan a session
  around one boot: submit the first job, let the worker come up, then run
  everything else back to back while it is warm (`idleTimeout` is 10 s, so
  "warm" means within seconds of the previous completion, not minutes).
- **`initializing` is not the same as wedged.** The endpoint fault C.12.12
  recorded shows **zero** workers in _every_ state with jobs queued. One worker
  `initializing` means placement worked and the host is pulling the image. Do
  not re-save the endpoint, do not rebuild, and do not resubmit — the queued
  jobs run when the boot finishes.
- **A queued job survives the client giving up.** The runner's 15-minute cap
  abandons the _wait_, not the job. Read it back later with
  `GET /v2/<endpoint>/status/<jobId>`; do not submit a second one, which just
  queues behind the first.

## 2d. Video extension (V.10.01)

Nothing here is needed for the image stack. Turn it on as a block — a
half-configured video endpoint is the dangerous state, not an unconfigured one,
which is why `dev-stack-preflight.mjs` skips every video check while
`RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID` is empty and binds all of them the moment it
is set.

- **`RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID`** — the second endpoint (V.03.04 prints
  it on creation; `scripts/isis/runpod-provision.mjs endpoint list` recovers it
  later). Every `motion`-category workflow routes here. There is no fallback to
  the image endpoint — a `motion` submit with this unset fails loud (V.06.03).
  Both endpoints run the same image off the same volume; the difference that
  matters is the card, and falling back would send a 28 GB expert to a 24 GB
  pool and bill the OOM (`comfyui.service.ts:967`).
- **`ISIS_COST_RATE_RUNPOD_VIDEO_USD_PER_SECOND=0.001647`** — **H200 141 GB flex
  ($5.93/h), not the H100's 0.001331.** The endpoint's `gpuIds` list includes
  H200, and a list means "any of these": the ledger cannot know which card ran,
  so it has to bill the dearest or it under-counts by 24% every time an H200 is
  assigned. (It carried 0.001331 until 2026-09-12 because the preflight's price
  table had no H200 row and reported it UNPRICED rather than guessing — correct
  behaviour that still left the rate wrong; the table now prices all seven
  enabled cards from §2f's measured tiers.) Copying the IMAGE rate here is the
  other mistake to fear: an H200 second costs ~3.8x a 5090 second, so the ledger
  would under-count by that factor and the budget would never fire. The
  preflight asserts this against the `comfyui-video` entry's `gpuIds` in
  `infra/runpod/endpoints/desired-state.json` and fails if the configured rate
  is below the priciest GPU that endpoint may be assigned (0.000486 if V.11.06
  moves Wan to L40S).
- **`ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6`** — a second ledger with its own
  Redis key (`:video`), separate from the image one. **A single video job can
  cost more than the entire image day budget:** a 5-second 720p Wan 2.2 clip is
  a few minutes of H100 time, and the image budget is $3/day total. The
  preflight caps this at 50 for the same reason.
- **Volume S3** (`RUNPOD_S3_ACCESS_KEY`, `RUNPOD_S3_SECRET_KEY`,
  `RUNPOD_VOLUME_ID=5bst2pgw7y` (the model cache; `ax0oj250j3` was deleted
  2026-09-13), `RUNPOD_S3_ENDPOINT=https://s3api-eu-ro-1.runpod.io`, and
  `BUCKET_ENDPOINT_URL` for the worker side). Inputs and outputs above the
  inline ceiling ride the volume rather than the request body. The preflight
  checks the two credentials by LENGTH only and never prints them, and refuses a
  `BUCKET_ENDPOINT_URL` whose host disagrees with `RUNPOD_S3_ENDPOINT` or whose
  path does not name the volume — pointing the worker's bucket anywhere else
  writes outputs where nothing reads them.

- **Retention sweep** (`ISIS_VOLUME_RETENTION_SWEEP_ENABLED`,
  `ISIS_VOLUME_RETENTION_DRY_RUN`, `ISIS_VOLUME_RETENTION_DAYS`,
  `ISIS_VOLUME_RETENTION_SWEEP_INTERVAL_MS`). The API can delete
  `inputs/<jobId>/` and `outputs/<jobId>/` objects past an age, so the volume
  does not get grown to store last month's test clips. It is **off unless
  enabled and a dry run even then** — set `ISIS_VOLUME_RETENTION_DRY_RUN=false`
  to actually delete, and read the dry-run log line first; it is the same line
  the real run prints. `models/` is not sweepable at any age and the refusal is
  in the client, not the config. By hand:
  `node scripts/isis/runpod-volume-s3-put.mjs sweep --days 7` (add `--yes` to
  delete).

**`ffprobe` is a convenience here, not a dependency.** The API reads a clip's
real duration, fps, frame count and dimensions out of the MP4 boxes itself
(`apps/isis/generation-api/src/services/mp4-probe.ts`), precisely because
`ffprobe` is silently absent on a slim image and a stack that depends on it
fails in a way nobody notices. Install it anyway for cross-checking a suspicious
clip by hand:

```bash
ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height,nb_frames,avg_frame_rate \
  -show_entries format=duration -of default=nw=1 out.mp4
# no output at all -> not installed; `brew install ffmpeg` (macOS),
# `apt-get install -y ffmpeg` (Linux dev server). Nothing in the API needs it.
```

Two honest caveats when comparing its numbers to the API's: `mp4-probe` reports
the AVERAGE fps over the track (frames / track duration), so a clip truncated by
`-shortest` muxing reads lower than its nominal rate — measured at 12.8 vs 16 on
such a file, while every untruncated clip (5/9/17/33/81 frames at 16 fps) agrees
exactly. And WebM is a different container (EBML): the parser returns null for
it rather than guessing, so an alpha `.webm` keeps whatever the graph declared.

### 2f. Stopping the video endpoint, and what happens when there is no GPU (V.03.05)

Endpoint **`2lezocymx2rgot`** (`isis-comfyui-video`), EU-RO-1, `workersMin 0` /
`workersMax 1`, 900 s execution cap.

**Kill switch — `workersMax` to 0.** Either the dashboard's per-endpoint button
(`runpod-kill-switch.service.ts`, target `video`) or:

```bash
node scripts/isis/runpod-provision.mjs endpoint update 2lezocymx2rgot --workers-max 0 --yes
node scripts/isis/runpod-endpoint-drift-check.mjs --key comfyui-video   # expect DRIFT on workersMax
```

A bare `runpod-endpoint-drift-check.mjs` with no `--key` now checks **every**
endpoint in the environment and says how many it checked; `--key` narrows it to
one.

**A paused endpoint queues NOTHING — it refuses the submit.** Measured against
the live video endpoint at `workersMax: 0` (V.11.04, 2026-09-12), `/run`
answers:

```json
{
  "status": 409,
  "title": "Conflict",
  "code": "ENDPOINT_PAUSED",
  "detail": "Endpoint is paused (max_workers=0). Set max_workers > 0 to accept work."
}
```

`/health` stayed at `inQueue: 0` with no worker in any state. So the switch is
stronger than "nothing new starts": there is no backlog waiting to stampede when
the ceiling goes back up, and nothing to cancel. The API surfaces it as
`RUNCOMFY_ENDPOINT_PAUSED` with the message "The RunPod endpoint is stopped
(workersMax is 0) and is accepting no work. Raise workersMax in the RunPod
console to resume.", and does **not** retry it — a paused endpoint stays paused
until a human raises it, so every retry would be another 409. (It arrived as
"Unexpected RunPod provider error." until this was fixed.)

Raising it again is deliberate and manual. Two more things measured 2026-09-12:

- **`/run` keeps answering `409 ENDPOINT_PAUSED` for ~30 s after you raise
  `workersMax` again.** REST reports the new value immediately. That lag is
  propagation, not the wedged-endpoint fault — wait half a minute and retry
  before diagnosing anything.
- A job that DID reach the queue before the pause **outlives the client that
  submitted it**. `runpod-catalog-run.mjs` gives up after 15 minutes; the job
  does not. Read it back with `GET /v2/<endpoint>/status/<jobId>` rather than
  submitting a second one, which only queues behind the first.

**No GPU available: the job waits, and then the API gives up. It does NOT fall
back to a smaller card.** That refusal is deliberate — a 28.6 GB fp16 Wan expert
that OOMs has already been billed for the load, so a "helpful" fallback to a 48
GB card converts a wait into a charge for nothing. The endpoint is pinned to
cards that all clear the 70.53 GB peak (three RTX PRO 6000 Blackwell variants,
three H100 variants, H200) and nothing smaller is enabled.

While the job sits `IN_QUEUE` nothing is billed — RunPod's `delayTime` is not
GPU time — so the cost of waiting is wall clock only. What ends the wait is the
queue/boot allowance from C.12.17 (`ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS`, default
**3 600 000 — one hour**), after which the API cancels the job as `retryable`
because nothing was spent.

**Every queue-side deadline is one hour as of 2026-09-12**, and they are a
different thing from the execution caps. Queue wait is unbilled, so a short
deadline buys nothing and throws away work that was about to run:

| Deadline                                | Value      | Where                                     |
| --------------------------------------- | ---------- | ----------------------------------------- |
| API queue/boot allowance, per attempt   | 1 h        | `ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS`       |
| Matrix runner give-up (cold start only) | 1 h        | `motion-live-proof.mjs`                   |
| Single catalog run                      | 1 h        | `runpod-catalog-run.mjs`                  |
| Orphan collector                        | 1 h        | `motion-proof-collect.mjs --deadline-min` |
| **Execution cap, image**                | **5 min**  | `ENDPOINT_EXECUTION_TIMEOUT_MS.image`     |
| **Execution cap, video**                | **15 min** | `ENDPOINT_EXECUTION_TIMEOUT_MS.video`     |

The last two bound BILLED GPU time and are the cost guard. Do not raise them to
match the queue ones.

**Supply in EU-RO-1 is the real constraint, and it is not what the console shows
you first.** Measured 2026-09-12, the same three tiers before and after
attaching the network volume:

| Tier            | Before volume (global) | After volume (EU-RO-1) |
| --------------- | ---------------------- | ---------------------- |
| 96 GB, $3.49/h  | High Supply            | **Low Supply**         |
| 80 GB, $4.79/h  | High Supply            | **Unavailable**        |
| 141 GB, $5.93/h | Medium Supply          | **Unavailable**        |

So the three-deep fallback is, in practice, a chain of one. All three stay
enabled because "Unavailable" is a point-in-time reading rather than a property,
but **plan on 96 GB or a wait**. Never size a RunPod endpoint from the labels
shown before the volume is attached — they describe datacenters you will not be
deployed in.

**Re-measured 18:20Z the same day, with every tier the console offers, because
V.11.01 could not get a card at all** (three attempts, `throttled`, `$0.00`
billed, and the console saying "There's no free GPU capacity for your current
selection"):

| Tier             | Enabled | Supply         | $/hr   |
| ---------------- | ------- | -------------- | ------ |
| 96 GB Pro        | 1st     | **Low Supply** | 3.49   |
| 80 GB Pro (H100) | 2nd     | Unavailable    | 4.79   |
| 141 GB (H200)    | 3rd     | Unavailable    | 5.93   |
| 80 GB (A100)     | 4th     | Unavailable    | 2.72   |
| 16 GB            | no      | Medium Supply  | 0.58   |
| 24 GB Pro        | no      | Low Supply     | 1.10   |
| 24 / 32 / 48 GB  | no      | Unavailable    | ≤ 1.75 |
| 180 GB Pro       | no      | Unavailable    | 8.64   |
| 288 GB Pro       | no      | Unavailable    | 10.65  |

**Every tier that can hold a 70.53 GB Wan expert is Unavailable except the 96 GB
Pro one, which is Low Supply.** Adding the two A100 80 GB variants (done the
same day, endpoint version 2) did not help: that tier is dry too. There is no
card selection that fixes this — the only ≥80 GB tier with any supply is the one
that was already first in priority. **If a clip has to run, the region is the
only lever left; see §2g.** Note also that a worker can reach `ready` and be
taken back before it picks up a job (observed 18:14:40Z → 18:15:00Z), so a
`ready` reading is not yet capacity.

Two practical consequences when supply IS found:

- Nothing is billed while a job waits or a worker is `throttled` — the console's
  billing panel read `$0.00` across all three attempts. Waiting costs wall clock
  only, and the C.12.17 queue-phase timeout retries it for free.
- The 10 s `idleTimeout` is dangerous in a low-supply region: lose the worker
  between two clips of a warm session and the next one rejoins the same queue
  with no guarantee of a card. Raise it for the duration of a multi-clip session
  (V.11.02) rather than paying for a second placement gamble.

### 2g-migration. Moving to US-CA-2 (in progress 2026-09-12)

> **Closed 2026-09-12 23:58Z — the stack stays in EU-RO-1.** User decision after
> the §2g-full-survey below: no region has proven supply of both card tiers,
> US-CO-1 is H200-only, and US-CA-2 never placed a worker. Everything outside
> EU-RO-1 was deleted in one pass (endpoints `eron7rmrmcg0rq`, `ajcpag53ed052j`,
> `ay4gcv868ein1u`; their templates plus the seven probe templates; volumes
> `sq5230axbc` US-CA-2 and `fj3f77pca3` US-CO-1). What remains is exactly the
> pre-migration state: volume `ax0oj250j3`, endpoints `2lezocymx2rgot` and
> `6jye7vngg4i685`. The recipe below is kept because it is correct for any
> future move — read the survey and the region-hunt rules first.

Approved after the probe below. **Nothing in EU-RO-1 is touched until a clip has
rendered in US-CA-2** — the 500 GB volume there is the only copy of 358 GB that
took hours to fetch, and a half-finished migration that deleted it would be
unrecoverable.

```bash
# 1. Volume in the new region. A volume PINS the datacenter, which is also why
#    the existing endpoints cannot follow it — new ones are required.
curl -s -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
  -d '{"name":"isis-chroma-models-us","size":500,"dataCenterId":"US-CA-2"}' \
  https://rest.runpod.io/v1/networkvolumes          # -> sq5230axbc

# 2. Loader pod. CPU-only ($0.06/h); V.02.04 forbids a GPU pod for this.
node scripts/isis/runpod-provision.mjs pod create-loader \
  --volume sq5230axbc --dc US-CA-2 --yes            # -> mf35u5m5k0c8xi

# 3. Watch it. The .part suffix is the loader resuming with HTTP Range.
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-2.runpod.io
node scripts/isis/runpod-provision.mjs volume ls sq5230axbc --dc US-CA-2

# 4. The verdict is the loader's own, not a size check: it verifies every
#    sha256 and exits 0 only if all 42 rows are "ok".
node scripts/isis/runpod-provision.mjs volume cat sq5230axbc .oshun/manifest-result.json --dc US-CA-2

# 5. Delete the pod the moment the pull is done — it bills while it lives.
node scripts/isis/runpod-provision.mjs pod delete mf35u5m5k0c8xi --yes
```

**A CPU pod cannot run the worker image.** `POST /pods` with `computeType: CPU`
rejects `containerDiskInGb` above 20, and both live templates for the video
image declare 30. A CPU pod created at 20 GB sat 13 minutes with `runtime: null`
(pull never finished) before being deleted. Anything that needs the image's own
filesystem — the build-time `object_info.json` in `/opt/oshun/build-manifest/`,
for one — has to run as a serverless GPU worker on the same volume.

**Set `workersStandby: 0` explicitly on the new endpoints.** Creating a
serverless endpoint through the REST API defaults it to 3 (and `workersMax` to
3), and a standby worker starts billing **with no job submitted** — see the
probe note below. The two EU endpoints carry `workersStandby: 1` and are not
billing for it only because EU-RO-1 cannot place a worker at all.

**Order of the cutover**, deliberately conservative: pull → verify all 42
sha256s → create both endpoints on the new volume → render one clip → only then
delete the EU volume and its endpoints. The EU stack keeps serving throughout.

**Everywhere the region is pinned.** Found by grep rather than memory, because a
missed one fails as "no GPU available" or a silently wrong provenance record
rather than as an error:

| File / setting                                                                  | Now                               | At cutover                                                                                  |
| ------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------- |
| `infra/runpod/volumes/isis-chroma-models.manifest.json` → `volume.dataCenterId` | `EU-RO-1`                         | `US-CA-2`                                                                                   |
| …same file → `volume.name`                                                      | `isis-chroma-models`              | `isis-chroma-models-us`                                                                     |
| `infra/runpod/endpoints/desired-state.json` → both `dataCenterId`               | `EU-RO-1`                         | `US-CA-2`                                                                                   |
| …both `networkVolumeId`                                                         | `ax0oj250j3`                      | `sq5230axbc`                                                                                |
| …both `endpointId`                                                              | EU ids                            | the new ids                                                                                 |
| …both `workersStandby`                                                          | not declared (RunPod default)     | **`0`, explicitly**                                                                         |
| Env recipe `RUNPOD_REGION`                                                      | `EU-RO-1`                         | `US-CA-2` — it is the provenance bundle's `region`, so a stale value mislabels every render |
| Env recipe `RUNPOD_S3_ENDPOINT`                                                 | `https://s3api-eu-ro-1.runpod.io` | `https://s3api-us-ca-2.runpod.io`                                                           |
| Env recipe `RUNPOD_VOLUME_ID`                                                   | `ax0oj250j3`                      | `sq5230axbc`                                                                                |
| Env recipe both endpoint ids                                                    | EU ids                            | the new ids                                                                                 |

The manifest is **shared** between the two volumes while both exist, so its
`volume` block is changed at cutover and not before — editing it early would
make the lint describe a volume that is still serving.

`RUNPOD_REGIONS` in `libs/isis/runpod-surface/src/endpoint-registry.ts` is an
AWS-style list (`us-east-1`, …) unrelated to RunPod's datacenter ids and needs
no change; `RUNPOD_REGION` is a free string read only for the provenance
`region`.

**The S3 listing returns intermittent HTTP 524.** Two tries in three failed
while the loader was writing. It is Cloudflare timing out on the listing, NOT an
empty volume — retry before believing a zero, and never report an error as 0 GB.

### 2g-region-hunt. US-CO-1 grants cards; US-CA-2 and EU-RO-1 do not (2026-09-12 23:2xZ)

**How to test a region, since the obvious routes do not work.** The console's
supply labels need a volume to pin the datacenter and are far too slow to read a
dozen times. GraphQL introspection is disabled.
`lowestPrice(input:{dataCenterId})` returns almost nothing per-DC and quotes
B200 at the same $5.98 everywhere — it is the POD market, not serverless. So
test **placement**, which is the ground truth anyway and is pure API:

1. Create a 10 GB volume in the datacenter (a volume is what pins it).
2. Create a throwaway endpoint on it with the cards you actually want, and a
   tiny public image so nothing waits on a pull.
3. Read `/v2/<id>/health` and count **`ready + idle + running`**.
4. `workersMax: 0`, then delete endpoint → then volume (a volume attached to a
   live endpoint returns 500).

**`initializing` is NOT evidence of a card.** It is equally what a worker shows
while WAITING for one — the Workers tab spells it out: "Waiting for GPU to
become available…". Only `ready`/`idle`/`running` proves a card was granted.
Counting `initializing` as success will tell you a starved region is healthy.

**Measured, same eight large cards offered to all four:**

| Datacenter  | Card granted?        | Evidence                                                                   |
| ----------- | -------------------- | -------------------------------------------------------------------------- |
| **US-CO-1** | **yes, repeatedly**  | `placed=2` at 23:21, 23:22 and 23:23                                       |
| US-GA-2     | once, not repeatable | `ready`/`idle` on the first read only                                      |
| US-CA-2     | **no**               | 1 h+ queued; the V.11.02 verification run gave up having never started, $0 |
| EU-RO-1     | **no**               | `initializing` all session, never placed                                   |

**The account caps at 5 max-workers across all endpoints**, so probes have to be
run in batches and torn down between them.

**One placement is not evidence.** US-CA-2 was migrated to on the strength of a
single worker appearing within a minute at 20:02Z, and it has not granted a card
since — the 358.30 GB now sits in a region that cannot serve it. US-CO-1 is
recorded here only because it held across three consecutive readings. Apply that
bar before moving anything again.

### 2g-full-survey. Every storage-capable datacenter, one form read (2026-09-12 23:50–23:53Z)

The question that should have been asked before any volume was created: **which
regions have the card tiers BOTH endpoints need?** Read from the console's
deploy form with exactly one datacenter selected and no volume, for all
seventeen datacenters GraphQL marks `storageSupport: true` — a volume pins the
region, so nowhere else can host the stack. Raw labels and prices:
`docs/agents/evidence/isis-chroma-runpod/region-supply-survey-2026-09-12T2350Z.json`.

Image needs 24 / 24 PRO / 32 PRO ($0.69 / $1.10 / $1.58 per hour). Video needs
80 PRO (H100, $4.79), 96 PRO (RTX PRO 6000, $3.49), 141 (H200, $5.93) or 180 PRO
(B200, $8.64). `—` is Unavailable.

| Region   | 24 GB | 24 PRO     | 32 PRO | 80 PRO | 96 PRO | 141 GB | 180 PRO |
| -------- | ----- | ---------- | ------ | ------ | ------ | ------ | ------- |
| EU-RO-1  | Low   | **Medium** | Low    | —      | Low    | —      | —       |
| US-CA-2  | Low   | Low        | —      | Low    | Low    | Low    | Low     |
| US-CO-1  | —     | —          | —      | —      | —      | Low    | —       |
| US-GA-2  | Low   | —          | —      | Low    | —      | Low    | —       |
| US-NC-2  | —     | —          | —      | —      | Low    | —      | Low     |
| EUR-IS-1 | Low   | —          | —      | —      | Low    | —      | —       |
| CA-MTL-3 | —     | Low        | —      | —      | Low    | Low    | —       |
| AP-JP-1  | —     | —          | —      | Low    | —      | Low    | —       |
| EU-FR-1  | —     | —          | —      | Low    | —      | Low    | —       |
| EUR-IS-3 | —     | —          | —      | Low    | —      | —      | —       |
| EUR-NO-1 | —     | Low        | Low    | —      | —      | —      | —       |
| EUR-NO-2 | —     | —          | —      | Low    | —      | —      | —       |
| US-IL-1  | Low   | **High**   | —      | —      | —      | —      | —       |
| US-MO-2  | Low   | —          | —      | —      | —      | —      | —       |
| US-TX-3  | —     | Low        | —      | —      | —      | —      | —       |
| CA-MTL-4 | —     | —          | —      | —      | —      | —      | —       |
| EU-NL-1  | —     | —          | —      | —      | —      | —      | —       |

**What it settles.**

- **US-CO-1 is an H200-only datacenter.** Three independent readings agree: this
  form, the earlier console table in §2f, and GraphQL
  `dataCenters.gpuAvailability` (which lists exactly one card there). The
  workers the region hunt saw placed can only have been H200s. So US-CO-1 is the
  most expensive place to run video short of B200 ($5.93/h — 1.7× the 96 GB PRO
  rate, 2.2× the A100 the EU clip actually billed at), and it has **nothing at
  all for the image endpoint**: the chroma stack could not follow the volume
  there. The region hunt measured "does a worker get placed" and never asked "on
  what card" — that is the gap this table closes.
- **No region has proven supply of both tiers.** Five carry labels for both
  (EU-RO-1, US-CA-2, US-GA-2, EUR-IS-1, CA-MTL-3); only EU-RO-1 has ever
  rendered anything for us, and US-CA-2 — the widest label coverage in the table
  — never placed a worker in three hours. **A Low label does not predict
  placement in either direction**: US-CA-2 had four Low tiers and placed
  nothing; US-CO-1 had one and placed repeatedly. Placement tests
  (§2g-region-hunt) remain the only ground truth, and they must now also record
  the card granted.
- **EU-RO-1 is the best image region on the board** — Medium on 24 PRO, the only
  Low-or-better 32 PRO besides EUR-NO-1, and every image job has rendered there.
  The image endpoint has no reason to move.
- **Candidates for video that are untested**: US-NC-2, CA-MTL-3 and EUR-IS-1
  list 96 GB PRO. Each needs the placement test with the video endpoint's exact
  card list before a byte is copied — the US-CA-2 migration is what happens when
  that step is skipped.

**Three volumes are billing right now** (EU-RO-1 `ax0oj250j3`, US-CA-2
`sq5230axbc`, US-CO-1 `fj3f77pca3`; $35/month each at 500 GB). US-CA-2 has never
served a job and US-CO-1 is empty; at most one of the two should survive the
decision.

**US-CO-1 has no pod market and no S3 API.** `POST /pods` rejects it (the REST
`dataCenterIds` enum predates the region; without the field, every CPU flavour
answers "no instances available"), and `s3api-us-co-1.runpod.io` does not
resolve. The only way to write to a volume there is a serverless worker in the
region — a CPU-serverless loader endpoint (`ay4gcv868ein1u`, template
`e2gcmznv3p`, `python:3.12-slim` running the C.02.06 loader behind a tiny
`runpod` handler that returns `manifest-result.json` as job output) was created
for that and parked at `workersMax: 0` pending this decision.

### 2h. The object-info snapshot comes out of the build log

`infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json` is the
worker image's own `/object_info`, reduced to what the graph validator reads,
and it is refreshed like this — no GPU, no pod, no S3:

```bash
git push origin HEAD:runpod-comfyui-slim          # RunPod rebuilds; smoke emits OBJINFO lines last
node scripts/isis/runpod-build-log.mjs --latest --endpoint 2lezocymx2rgot   # wait for COMPLETED
node scripts/isis/runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot        # verifies chunks + sha256, writes the file
npx vitest run --root libs/isis/workflows graph-validator                    # every golden against it
```

Why the log: `getBuildLogs` takes only `buildId` and `startTime` and returns the
newest 500 lines, so the smoke prints the snapshot as the very last thing (75
lines of 1600 chars for 1071 classes). A CPU pod cannot hold the image
(`containerDiskInGb` ≤ 20, the image needs 30), US-CO-1 has no S3 API, and a GPU
worker in EU-RO-1 waited over an hour without a card. The root `.dockerignore`
excludes `infra`; the manifest is re-included by name so the smoke can shadow
the volume — a new file under `infra/` that the build needs must be re-included
the same way.

### 2g-supply. Both regions fluctuate; neither is reliably warm

The honest comparison, each read off the endpoint's own GPU configuration list:

| Tier                    | EU-RO-1 @ 18:20Z | US-CA-2 @ 22:55Z |
| ----------------------- | ---------------- | ---------------- |
| 96 GB Pro, $3.49/h      | Low Supply       | **Unavailable**  |
| 80 GB Pro H100, $4.79/h | Unavailable      | **Low Supply**   |
| 141 GB H200, $5.93/h    | Unavailable      | **Low Supply**   |
| 80 GB A100, $2.72/h     | Unavailable      | Unavailable      |
| 180 GB B200, $8.64/h    | Unavailable      | **Low Supply**   |

US-CA-2 has **two** of the enabled large tiers at Low Supply where EU-RO-1 had
one, plus B200 — which is why the move is defensible. But at 22:55Z US-CA-2 was
ALSO answering "There's no free GPU capacity for your current selection" with
two jobs queued. **Neither region is reliably warm, and a single placement is
not evidence that one is.** The probe that triggered this migration got a card
in US-CA-2 in under a minute at 20:02Z; the same selection could not get one
three hours later. Read the per-tier table before concluding anything about a
region.

**`Initializing` does not mean "pulling the image".** It is also what a worker
shows while it waits for a card it has not been given. The Workers tab says
which — "Waiting for GPU to become available…" — and the Logs tab stays empty
("logs will appear here once they're ready"). Do not infer a pull from the
status string.

**B200 is enabled on the US video endpoint** (2026-09-12) because it was the
only large tier showing supply there and it is the one plausible shot at
`wan22-animate`, which died at the floor of the caps and may never have had a
card big enough. It costs $8.64/h, so it is now the dearest card either endpoint
can be assigned and **the video ledger rate rises to $0.0024/s** — every video
estimate goes up about 46%, which makes the budget gate more conservative, not
less.

### 2g-billing. A first-time image pull in a new region is NOT billed

Read off the endpoint's own Billing panel while a worker had been `initializing`
for 27 minutes on its first-ever pull in US-CA-2:

> **No charges in the past 24 hours** · Spend rate **$0.00000 / second** · Total
> billed **$0.00**

So the picture is now complete and consistent with V.11.01's account evidence:
**RunPod bills execution. Not queue wait, not worker boot, not image pull.**
That is why our ledger charging `delayTime` overstated a clip by 4x, and it is
why a cold region is worth waiting out rather than abandoning — a 27-minute
first pull costs nothing but wall clock.

**Check the panel rather than reasoning about it.** The endpoint page shows
`Spend rate` and `Total billed` live, and it answered in one page-load a
question that is otherwise unanswerable from the API.

### 2g-probe. US-CA-2 has capacity EU-RO-1 does not (measured 2026-09-12)

The migration question is no longer speculative. A 10 GB volume in **US-CA-2**
with a throwaway endpoint carrying **the identical nine cards** as the video
endpoint was handed a worker **inside a minute, with no jobs queued at all**,
while EU-RO-1 at the same instant sat `throttled` with two jobs waiting. Same
card ids, opposite outcome: the constraint is the REGION.

Caveats, because one reading is one reading: the probe ran a tiny `runpod/base`
image, so it measured **card availability and not boot time** — a 288 GB volume
and a multi-GB image still have to be pulled wherever we go. US datacenters that
support network volumes: `US-CA-2`, `US-CO-1`, `US-GA-2`, `US-IL-1`, `US-MO-2`,
`US-NC-2`, `US-TX-3`.

**Two traps the probe walked into, so you do not have to.**

1. **Creating a serverless endpoint through the REST API starts a worker
   immediately, with no job submitted.** RunPod defaults `workersStandby` to 1
   (3 when `workersMax` is left unset), and **`workersStandby` cannot be set at
   all through REST** — it is rejected as an unknown key on both create and
   patch. So every endpoint you create begins billing at once.

   **`PATCH {"workersMax": 0}` does stop it, within about 40 seconds.** An
   earlier note here said it did not and that only deleting the endpoint worked;
   that was wrong — the probe was deleted ~20 s after the patch, before the
   worker had wound down. Measured on `isis-comfyui-video-us`: `initializing` at
   t+20 s, no workers at t+40 s.

   **So the recipe is: create → immediately `workersMax: 0` → raise it to 1 only
   when you are ready to submit.** That costs well under a minute of one worker
   instead of however long you take to notice.

2. **Both real endpoints carry `workersStandby: 1`.** They are not billing for
   it today only because EU-RO-1 cannot place a worker at all. That is
   starvation suppressing a standing cost, not a safe configuration — re-check
   it before moving to a region that can actually serve us.

### 2g. The image endpoint was rebuilt, and the region option if GPUs run out

**Endpoint `6jye7vngg4i685`** (was `6l1f3tku7z2zoy`, deleted 2026-09-12),
template `vfl3iu1dr6`, container disk **30 GB**.

**Why it was rebuilt: container disk.** The old endpoint's template allocated 20
GB. When V.01.01 added the six video node packs and their dependencies (cupy,
triton, timm, pycocotools, the controlnet_aux annotators) the image outgrew it,
and from Release #10 onward **no worker ever started** — placement succeeded,
the container was never created, and the worker sat `initializing` indefinitely
with **completely empty logs**. Raising the disk on the old endpoint did not
recover it; a rebuild did, and the first job on the new one ran normally.

If a worker ever hangs in `initializing` with no log output again, **check the
container disk against the image size first.** It is the one field neither API
exposes for a git-integration template — `GET /v1/templates/<id>` 404s and
GraphQL `podTemplates` omits it — so it is console-only, under Manage → Edit
endpoint.

**Region is NOT the usual cause, but it is the fallback.** Serverless supply per
region, measured in the console's endpoint-create form on 2026-09-12 with
exactly one datacenter selected (labels are point-in-time and moved measurably
within the same session):

| Region                | 24 GB | 24 PRO   | 32 PRO  | 48 PRO | 80 PRO | 96 PRO | 141 GB |
| --------------------- | ----- | -------- | ------- | ------ | ------ | ------ | ------ |
| **EU-RO-1** (current) | Low   | Low      | **Low** | —      | —      | Low    | —      |
| US-CA-2               | —     | Low      | —       | —      | Low    | Low    | Low    |
| US-IL-1               | Low   | **High** | —       | Low    | —      | —      | —      |
| US-GA-2               | Low   | —        | —       | —      | Low    | —      | Low    |
| US-MO-2               | Low   | —        | —       | —      | —      | Low    | —      |
| US-TX-3               | —     | Low      | —       | Low    | —      | —      | —      |
| US-NC-2               | —     | —        | —       | —      | —      | Low    | —      |
| US-CO-1               | —     | —        | —       | —      | —      | —      | Low    |

Read it this way: **EU-RO-1 is the only region with 32 GB PRO**, which matters
because Chroma1-HD (17.8 GB) plus T5-XXL (9.8 GB) is a tight fit in 24 GB.
**US-CA-2 is the only region covering both tiers** — image and all three video
sizes — so it is the migration target if one is ever needed. **US-IL-1 has the
only _High_ reading anywhere** (24 GB PRO) but nothing for video.

**Trigger for migrating (user decision 2026-09-12): only if `no GPU available` /
sustained `throttled` starts recurring.** It is not worth doing pre-emptively.
The move is: create a 500 GB volume in the target region, run the C.02.06 loader
against the same manifest (~$0.06 of CPU pod, ~1 hour for 358 GB), verify all 42
sha256s, rebuild both endpoints there, and only then delete the EU-RO-1 volume.
Keep both volumes until the new one verifies — the overlap is about $0.05/h.

**Two API traps if you do migrate.** `NVIDIA RTX PRO 4500 Blackwell` — the card
that ran every successful C.11 job — is **not accepted by the serverless REST
API**, and PATCHing it sets `gpuTypeIds` to `null` rather than erroring, leaving
the endpoint with no GPUs at all. It is only reachable as the pool id
`BLACKWELL_32`, which REST rejects, and the console's "32 GB PRO" tier maps to
`ADA_32_PRO` instead. Also, REST reports a **stale** `gpuTypeIds` for an
endpoint whose GPUs were changed through the pool API — trust the console.

## 2b. Dev credential

Provider routes (`/api/v1/jobs/runpod/*`, `/ready` detail) need a bearer token
with an operator role; the API has no API-key validator wired. Add a throwaway
`JWT_SECRET` (≥ 32 chars, e.g. `openssl rand -hex 32`) to the env recipe, then:

```bash
TOKEN=$(node scripts/isis/mint-dev-jwt.mjs --hours 12)   # HS256, admin role, project_isis_dev
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:4010/api/v1/jobs/runpod/cost/today
```

Paste the same token into the dashboard's Login page (bearer token field).

## 3. Preflight (refuses to continue on any incoherence)

```bash
node scripts/isis/dev-stack-preflight.mjs
node scripts/isis/validate-chroma-endpoint-config.mjs --require-image
```

## 4. Generation API (port 4010)

```bash
memory_pressure | tail -1
( cd apps/isis/generation-api && pnpm dev ) > /tmp/isis-gen-api.log 2>&1 &
echo $! >> /tmp/claude-<session>.pids
until curl -sf http://127.0.0.1:4010/health >/dev/null; do sleep 2; done
curl -s http://127.0.0.1:4010/health | head -c 400
curl -s http://127.0.0.1:4010/api/v1/jobs/runpod/cost/today
```

Log lines to expect: `Inline job runner started`,
`Workflow catalog bootstrap complete … includeChromaCatalog: true`,
`RunPod adapter initialized`.

tsx servers do not pick up source edits — restart after any change.

## 5. Dashboard (Vite, port 5180)

```bash
( cd apps/isis/web && VITE_ISIS_GENERATION_API_URL=http://127.0.0.1:4010 pnpm dev ) \
  > /tmp/isis-web.log 2>&1 &
echo $! >> /tmp/claude-<session>.pids
```

Open http://127.0.0.1:5180 → Login accepts a pasted bearer token or API key;
with `NODE_ENV=development` the API allows anonymous access.

The **Workflows** page is backed by the separate workflow-registry service
(user-authored workflows, port 3003), which does not know the catalog, so the
Chroma flows are submitted from **Job Submission Studio**: type `text-to-image`
(or `image-to-image` / `inpainting`), prompt, and in **Extra parameters
(JSON)**:

```json
{
  "workflowId": "chroma-txt2img",
  "checkpoint": "chroma1-hd",
  "aspect": "square_1024"
}
```

Every catalog input travels under its own name (`checkpoint`, `aspect`, `cfg`,
`steps`, `seed`, `denoise`, `model`, `text_prompt`, `grow_mask_by`,
`confidence_threshold`); image inputs (`source_image`, `mask_image`) take a
`data:image/png;base64,…` string. The **Jobs** page shows the run (claimed by
`inline-<pid>`) and, once completed, the output thumbnail with seed, checkpoint,
measured cost and execution time; the **RunPod** page shows spend.

## 5b. Worker outputs to Hetzner Object Storage (C.12.06 / V.12.05, decided 2026-09-13)

The stock worker's uploader names its bucket after the month and cannot sign for
a non-AWS host, so the image carries an `oshun_uploader.py` (imported at
start-up through an `oshun_uploader.pth`) that replaces `rp_upload.upload_image`
whenever `BUCKET_ENDPOINT_URL` is set (see
`docker/runpod/comfyui-slim/oshun_uploader.py`). Outputs then land in Hetzner
Object Storage and come back to the API as presigned URLs, which the API fetches
into its own output store — so the browser still sees one kind of URL.

Set these on BOTH endpoints in the RunPod console (Manage → Edit endpoint →
Environment variables; a git-integration template's env cannot be edited through
either API). The two secrets come from
`~/Desktop/workspace/drawer/env-master.env` (`HETZNER_APP_DEV_S3_ACCESS_KEY`,
`HETZNER_APP_DEV_S3_SECRET_KEY`); copy the values, never commit them.

| Key                         | Value                                     |
| --------------------------- | ----------------------------------------- |
| `BUCKET_ENDPOINT_URL`       | `https://hel1.your-objectstorage.com`     |
| `BUCKET_ACCESS_KEY_ID`      | the Hetzner access key                    |
| `BUCKET_SECRET_ACCESS_KEY`  | the Hetzner secret key                    |
| `BUCKET_NAME`               | `app-dev-server`                          |
| `BUCKET_KEY_PREFIX`         | `isis/outputs`                            |
| `BUCKET_REGION`             | `hel1`                                    |
| `COMFY_LOG_LEVEL`           | `INFO` (the DEBUG default floods the log) |
| `COMFY_POLLING_MAX_RETRIES` | `1500` (image) / `3600` (video)           |
| `COMFY_POLLING_INTERVAL_MS` | `250`                                     |

Objects are keyed `isis/outputs/<jobId>/<8 hex><ext>`. The API side needs
`ISIS_WORKER_OUTPUT_FETCH_HOSTS=hel1.your-objectstorage.com` (§2); any other
host in a worker's `s3_url` is refused with the reason recorded on the output.
The worker log prints `oshun uploader: rp_upload.upload_image -> bucket …` at
start when the patch is active, which is the check that the env reached it.

## 5c. Model library and cache (option B, tracker L.00–L.07, decided 2026-09-13)

**Where weights live.** The source of truth is the Hetzner library: bucket
`app-dev-server` on `https://hel1.your-objectstorage.com`, prefix
`isis/models/`, key = prefix + the manifest row's `destPath`
(`manifest.library`). The RunPod network volume is a **cache** of it
(`manifest.cache`, 400 GB, `isis-model-cache`): workers read from the cache, the
cache is filled from the library. Pinned families (`manifest.families[].pinned`:
the Chroma stack with its LoRAs, both Wan 2.2 expert pairs, the Wan shared
encoders, SAM 3, BiRefNet, FlashVSR, GIMM-VFI, Real-ESRGAN — 212.26 GB) are
never evicted; every other family stays until space is needed and then the
family used longest ago leaves first. Shared files (umt5, the Wan VAE,
clip_vision_h) leave only when no remaining family needs them, and a family a
queued or running job leases is never evicted.

**Adding a model.** (1) A manifest row with `families` (lint
`node scripts/isis/runpod-volume-manifest-lint.mjs --strict` must stay green,
including the cache invariants it prints); (2) if a workflow uses it, add the
family to that workflow's `requires_families` (`model-families.spec.ts` fails
otherwise); (3) put the bytes in the library — `library-pull` streams from
Hugging Face or Civitai with no local copy:

```bash
node scripts/isis/runpod-provision.mjs pod create-loader --manifest <manifest-with-the-new-rows> \
  --mode library-pull --dc EU-RO-1 --parallel 8 --dry-run      # keys show as <n chars>
node scripts/isis/runpod-provision.mjs pod create-loader ... --yes
```

(4) the cache picks it up the first time a job needs its family. Nothing is
copied to the cache by hand.

**Loader modes** (`LOADER_MODE`, `scripts/isis/runpod-volume-loader.py`):
`library-seed` (volume → library), `library-pull` (source → library),
`library-verify` (download and hash every row, write `verifiedAt`),
`library-probe` (L.00.01 transfer measurement), `cache-sync` (a plan of
evictions and pulls, applied to the mounted cache). Every mode writes
`/runpod-volume/.oshun/<mode>-result.json` (read it back with
`runpod-volume-s3-put.mjs` against the volume's S3 API) and prints a `RESULT`
line; delete the pod afterwards — it sleeps an hour for that purpose and bills
$0.06/h meanwhile. The two Hetzner keys are read from the env names in
`manifest.library.auth` and injected as pod env; `--dry-run` redacts them.
Single PUTs up to 512 MiB carry the manifest sha256 as the signed payload hash
(the server refuses wrong bytes); above that, multipart with a per-part hash and
a whole-file hash computed while reading — a mismatch aborts the upload.
Hetzner's multipart works (measured 2026-09-13); RunPod's volume S3 multipart
does not, which is why nothing uploads to the volume over S3 above 4 GiB.

**Warming a session.** Before a session that will use unpinned families
(LTX-2.3, Animate, VACE, the controlnet_aux preprocessors, RMBG-2.0), warm them:
`POST /api/v1/runpod/cache/sync` with
`{"workflows": ["ltx23-av"], "dryRun": true}` shows the pulls, the evictions and
the estimated wait; the same body without `dryRun` starts the sync (one per
volume at a time; it bills a `cache_sync` line at the CPU pod rate, and the kill
switch and budget guardrail apply). The dashboard's RunPod page has the same
"Warm for session" picker.

**`409 models_not_cached`.** A job whose families are not cached is refused at
submit, before any billing, with the families, their bytes, the estimated wait
and the exact sync call to make. With `ISIS_RUNPOD_CACHE_AUTOSYNC=true` the job
is held as `awaiting_cache` and the sync starts for it instead. When no cache
volume is configured (`ISIS_RUNPOD_CACHE_VOLUME_ID` and
`manifest.cache.volumeId` both unset) the gating is off and jobs run against the
old full volume.

**Pinning or unpinning a family.** Flip `pinned` in `manifest.families`, run the
lint (the invariants must still hold at the cache size), commit. Pinning does
not pull anything by itself; the next sync that plans for it keeps it.

**Licences (A.04.04).** Every weight's licence, territory, acceptable-use
clause, Civitai flags, content policy and distribution is in
[model-licences.md](./model-licences.md), generated from the manifest by
`node scripts/isis/model-licence-register.mjs` (its test fails when the file
drifts from the manifest). Adding a model edits the manifest row's licence
string and regenerates the register in the same commit; the exclusions table
names what must never be added.

**Outputs retention and the storage alarm (L.07.02).** Worker outputs under
`isis/outputs/` share the bucket, and the account's included allowance, with the
library. The API's Hetzner storage guard sweeps outputs older than
`ISIS_HETZNER_OUTPUTS_RETENTION_DAYS` (7) every
`ISIS_HETZNER_STORAGE_GUARD_INTERVAL_MS` (6 h), then sums every bucket the
credentials list and raises `storage.hetzner.usage_high` at
`ISIS_HETZNER_USAGE_ALARM_GB` (900). The library prefix is refused by the S3
client itself, whatever the env says. Every over-the-line check logs
`ops.hetzner_storage.usage_high` at error level; the webhook goes out once per
UTC day with event id `storage.hetzner.usage_high:<date>`. The same two jobs
from a shell:

```
node scripts/isis/runpod-volume-s3-put.mjs hetzner-usage            # exit 2 when over the line
node scripts/isis/runpod-volume-s3-put.mjs hetzner-sweep --days 7   # add --yes to delete
```

**Verifying the library.** `library-verify` over the manifest downloads and
hashes every row. The old 1024 GB volume `ax0oj250j3` was deleted on 2026-09-13
at 18:13Z on the user's explicit go (L.06.05), after both endpoints were
repointed to `5bst2pgw7y` and its loader result files were copied to
`docs/agents/evidence/isis-chroma-runpod/l-results/`. The library is now the
only full copy of the weights; a lost cache volume is rebuilt by syncing
families from it.

## 6. Submitting from the API instead (mask-only flows, scripts)

```bash
curl -s -X POST http://127.0.0.1:4010/api/v1/jobs -H 'Content-Type: application/json' -d '{
  "type": "text-to-image",
  "prompt": "a red bicycle leaning on a white wall, morning light",
  "parameters": { "workflowId": "chroma-txt2img", "checkpoint": "chroma1-hd", "aspect": "square_1024", "seed": 42 }
}'
curl -s http://127.0.0.1:4010/api/v1/jobs/<jobId>
```

A refusal with `RUNPOD_DAILY_BUDGET_EXCEEDED` (429) or
`RUNPOD_COST_LEDGER_UNAVAILABLE` (503) is the cost gate working, not a bug.

To run one catalog workflow straight at the endpoint, bypassing the API (this is
what the C.11 matrix used), it is **`npx tsx`, not `node`** — the script imports
`catalog-renderer.ts` at runtime and that module's extensionless imports are
unresolvable to plain Node ESM, which fails as
`Cannot find module .../expression-evaluator`:

```bash
npx tsx scripts/isis/runpod-catalog-run.mjs matting-remove-background \
  --image source_image=docs/agents/evidence/isis-chroma-runpod/chroma-txt2img-0-9d49c5b4.png \
  --set model=rmbg-2.0 --out /tmp/run-out
```

It needs `RUNPOD_API_KEY` and `RUNPOD_COMFYUI_ENDPOINT_ID` only — no dev stack.
It gives up after 15 minutes in `IN_QUEUE`, which on this endpoint is now a real
possibility: see §2e.

### 6c. Getting a failed job's worker log (it is a stream, not a document)

Three failures on 2026-09-12 — `wan22-animate` killing ComfyUI,
`flashvsr-upscale` returning a bare `Job processing failed`, and `ltx23-av`
before its cause was found in the graph — were all opaque for the same reason,
and on 2026-09-13 that reason turned out to be **a bug in the capture code**,
not the platform.

`GET https://api.runpod.io/v2/serverless/<endpoint>/workers/<workerId>/logs`
answers `content-type: text/event-stream`: `id:` plus `data:` frames, one JSON
object each (`{"source":"system"|"container","line":…,"ts":…}`), replayed from
the worker's first line at roughly eight frames a second and then held open for
live lines. It never closes. Both harnesses did `await response.text()` on it,
which can never resolve, and their 20 s timeout then wrote "log fetch timed out"
into every failure row. Measured live: `?tail=N` is honoured (the last N lines,
then live); `Last-Event-ID` and `?since=` are not; once the worker is recycled
the URL answers **404 "worker not found"** in about three seconds — that part of
the old explanation was right, and it still means a log is a race you can lose.

The reader is `scripts/isis/runpod-worker-log.mjs`: it asks for the tail,
collects frames until the stream goes quiet for 2.5 s or a 20 s cap lands, and
writes plain `<ts> <source> <line>` lines. A capped read is marked truncated
rather than passed off as complete. Both `runpod-catalog-run.mjs` and
`motion-live-proof.mjs` call it the moment a job comes back `FAILED`.

```bash
node scripts/isis/runpod-worker-log.mjs --endpoint "$RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID" --worker <workerId> --tail 400
node scripts/isis/runpod-worker-log.mjs --endpoint "$RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID" --worker <workerId> --out docs/agents/evidence/isis-chroma-runpod/<run>/worker-<workerId>.log
```

The worker id is on the job's status response (`workerId`) and on
`GET /v2/serverless/<endpoint>/workers`. `status.output.errors[]` remains the
first place to look — it carried `ltx23-av`'s
`Node 8 (LTXVImgToVideoInplace): required_input_missing 'image'` with no log at
all — but a `Job processing failed` with nothing in `errors[]` is exactly the
case the log exists for.

## 6b. Reading a RunPod build log (when a git-integration build fails)

The builds API tells you a build failed and almost nothing else:

```bash
curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \
  "https://api.runpod.io/v2/serverless/$RUNPOD_COMFYUI_ENDPOINT_ID/builds" | jq '.[0]'
# -> "Command '['/usr/bin/docker', 'buildx', 'build', ...]' returned non-zero exit status 1."
```

There is no `/logs` route under it — every shape 404s. Use:

```bash
node scripts/isis/runpod-build-log.mjs --latest          # the most recent build
node scripts/isis/runpod-build-log.mjs <buildId>          # a specific one
node scripts/isis/runpod-build-log.mjs --latest | grep -nE "ERROR|IMPORT FAILED|Traceback|error:"
```

It calls the same `getBuildLogs` GraphQL operation the console uses, with the
ordinary API key, and reverses the result — the API returns newest-first, and a
build log reads oldest-first. The first real error is near the top; the generic
docker summary is at the end and is never the reason.

**Do not skip this and change the Dockerfile on a hypothesis.** Two builds were
spent that way on 2026-09-12: the guess was `import cupy` on a GPU-less builder,
the reality was a dangling symlink breaking ComfyUI-SAM3's `os.makedirs`, and
the log said so plainly the whole time. It is the same lesson as C.12.12.

**If you try the console instead:** a RunPod console screenshot can come back
completely blank while the page is fully rendered and interactive. Check
`document.body.innerText.length` before concluding the page is broken. The log
panel is also virtualised, so scraping the DOM only ever gets the ~15 lines
currently on screen.

## 7. Teardown (every session)

```bash
for p in $(cat /tmp/claude-<session>.pids); do kill $p 2>/dev/null; done
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(4010|5180)'      # must be empty
rm -f /tmp/isis-chroma-env.sh
node scripts/isis/runpod-provision.mjs account            # pods 0, spend/h = volume only
```
