Eve library

Runbook: Isis Chroma on RunPod — local dev stack

Runbook

Revision 1

On this page

Tracker: ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md (C.10). Evidence: docs/agents/isis-chroma-runpod-evidence.md. Design: ADR-0005.

The generation API runs on this machine; the GPU work runs on the RunPod serverless endpoint (max one worker) against the network volume. Nothing here needs a local GPU.

0. Machine rules (CLAUDE.md, binding)#

  • memory_pressure | tail -1 before every heavy start; below 35 % free → tear down first.
  • One Vite/tsx server per session. Track PIDs in /tmp/claude-<session>.pids.
  • Servers die with the session: lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(4010|5180)' must be empty at the end.

1. Infra#

bash
docker compose -f docker/docker-compose.dev.yml up -d postgres redis
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(5432|6379)\b'   # both must listen

Postgres is postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev. The jobs table is created on first boot (CREATE TABLE IF NOT EXISTS jobs). Redis backs the daily USD ledger. MinIO is not needed until C.07 lands (outputs travel as data URLs inside the job record until then).

2. Env recipe (throwaway, never committed)#

Keys come from ~/Desktop/workspace/drawer/env-master.env; copy by name, verify by length only, delete the file at session end.

bash
S=/tmp/isis-chroma-env.sh
{
  grep -E '^(RUNPOD_API_KEY|RUNPOD_S3_ACCESS_KEY|RUNPOD_S3_SECRET_KEY|CIVITAI_API_KEY|GHCR_PULL_TOKEN)=' \
    ~/Desktop/workspace/drawer/env-master.env | sed 's/^/export /'
  echo "export JWT_SECRET=$(openssl rand -hex 32)"   # throwaway, never in env-master; see §2b
  cat <<'EOF'
export NODE_ENV=development
export PORT=4010
export BASE_URL=http://127.0.0.1:4010
export DATABASE_URL=postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev
export REDIS_URL=redis://127.0.0.1:6379
export ISIS_QUEUE_PRODUCER_ENABLED=false        # no BullMQ consumer exists locally
export ISIS_INLINE_JOB_RUNNER=true              # the API runs its own jobs (C.06.06)
export COMFYUI_PROVIDER=runpod
export ISIS_RUNPOD_WORKER_KIND=worker-comfyui   # API renders catalog → bare graph
export RUNPOD_COMFYUI_ENDPOINT_ID=6jye7vngg4i685   # rebuilt 2026-09-12 (V.03.04 note); see §2g
export RUNPOD_REGION=EU-RO-1                    # provenance ledger `region` (the volume's datacenter)
# export ISIS_PROVENANCE_ALLOW_UNWATERMARKED_IMAGES=true  # C.12.13 narrowed this
#   and OFF is now the right default. Images are watermarked for real; this flag
#   only decides what happens to the occasional output the stage could not mark
#   (off = that job's result is refused at completion, the GPU time having
#   already been spent). It no longer gates submission.
export ISIS_COST_RATE_RUNPOD_USD_PER_SECOND=0.000439   # RTX 5090 flex ($1.58/h) — the priciest
#   card the rebuilt endpoint accepts, NOT the 4090's 0.000306. A gpuIds list means "any of
#   these" and the ledger cannot know which card ran, so it has to bill the dearest one:
#   under-billing is the direction that stops the budget firing. The preflight fails on this.
export ISIS_RUNPOD_DAILY_BUDGET_USD=3
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=true
export ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD=10
export ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD=25
export ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW=3
export ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS=600
export ISIS_RUNPOD_BUDGET_FAIL_CLOSED=true
export ISIS_COST_RATE_DEFAULT_USD_PER_SECOND=0.000306
# export ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS=3600000  # C.12.17; default 3600000 (1 hour)
#   The UNBILLED queue + cold-boot allowance, held separately from the GPU
#   execution cap so a slow image pull cannot get a rendering job cancelled and
#   re-billed. **Raised from 10 min to 1 HOUR on 2026-09-12.** Ten minutes was
#   set when we thought a cold start's delayTime was billed, so waiting looked
#   expensive; V.11.01 measured that it is not (ledger $1.20, account $0.29).
#   Waiting is free, and EU-RO-1 needed FOUR attempts to place one clip — twice
#   a card arrived within a minute of a shorter deadline. Lower it (e.g. 45000)
#   for the kill-switch drills C.11.12 / V.11.04, which measure exactly this
#   wait; restore afterwards. A zero or malformed value is ignored, not obeyed.
export VITE_ISIS_GENERATION_API_URL=http://127.0.0.1:4010
export CORS_ORIGINS=http://127.0.0.1:5180   # the API rejects the dashboard's XHR without it
# export ISIS_RUNPOD_BUDGET_ALARM_PROJECT_ID=<project id>   # C.04.06: project whose webhook
#   subscriptions receive runpod.budget.warning / .exhausted. Unset = alarms log only.
#
# --- Video extension (V.00.02). Commented out until V.03.04 creates the second
# --- endpoint: an unset id is a loud failure, an empty one is a silent
# --- submission to nowhere. Uncomment the block as a whole, id first.
export RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID=2lezocymx2rgot   # V.03.04, created 2026-09-12; `motion` workflows route here
export ISIS_COST_RATE_RUNPOD_VIDEO_USD_PER_SECOND=0.001647   # H200 141 GB flex ($5.93/h) — the
#   dearest card in the EU endpoint's list (user decision 2026-09-13; 0.002400
#   was the deleted US endpoint's B200). A gpuIds list means "any of these" and
#   the ledger cannot know which card ran, so it bills the dearest; the card that
#   usually runs (RTX PRO 6000 Blackwell) settled at $0.00105/s on 2026-09-13.
# --- Model cache (option B, L.04). Leave ISIS_RUNPOD_CACHE_VOLUME_ID unset until L.06.01 creates the
# --- 400 GB cache volume: unset, cache gating is off and jobs run against the old full volume.
# export ISIS_RUNPOD_CACHE_VOLUME_ID=<isis-model-cache volume id>
export ISIS_RUNPOD_CACHE_SYNC_MB_PER_SECOND=40     # MiB/s; L.06.02 median with three transfers sharing Hetzner (about 75 alone; the L.00.01 probe read 20.8)
# export ISIS_RUNPOD_CACHE_AUTOSYNC=true            # hold jobs as awaiting_cache instead of 409
# --- Hetzner storage guard (L.07.02): outputs retention + the 900 GB account alarm. Off unless enabled,
# --- and the sweep is a dry run until the second line says false. Needs the HETZNER_APP_DEV_* names.
# export ISIS_HETZNER_STORAGE_GUARD_ENABLED=true
# export ISIS_HETZNER_OUTPUTS_RETENTION_DRY_RUN=false   # only after a dry-run log line looks right
# export ISIS_HETZNER_STORAGE_ALARM_PROJECT_ID=<ops project>   # falls back to ISIS_RUNPOD_BUDGET_ALARM_PROJECT_ID
export ISIS_WORKER_OUTPUT_FETCH_HOSTS=hel1.your-objectstorage.com   # C.12.06: hosts the API may
#   fetch presigned worker outputs from (the bucket the worker's uploader writes
#   to). Unset = only the RunPod volume's own S3 URL is accepted.
# export ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6      # second ledger key, separate from the image one
# export ISIS_RUNPOD_VIDEO_INPUT_INLINE_MAX_BYTES=  # V.05.01 measures it; above this, inputs go to the volume over S3
# export RUNPOD_VOLUME_ID=5bst2pgw7y                # the model cache volume, the only one since 2026-09-13
# export RUNPOD_S3_ENDPOINT=https://s3api-eu-ro-1.runpod.io   # SigV4 host for that bucket
EOF
} > "$S"; chmod 600 "$S"; source "$S"
echo "RUNPOD_API_KEY len=${#RUNPOD_API_KEY} S3 len=${#RUNPOD_S3_ACCESS_KEY}/${#RUNPOD_S3_SECRET_KEY} civitai len=${#CIVITAI_API_KEY} ghcr len=${#GHCR_PULL_TOKEN} jwt len=${#JWT_SECRET} endpoint len=${#RUNPOD_COMFYUI_ENDPOINT_ID} video-endpoint len=${#RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID}"

2c. What the first live run (2026-09-11) made mandatory#

  • Consent id on every job. The provenance ledger (active whenever DATABASE_URL is set) refuses to complete a generation without consentId; the API now refuses such jobs at submit, before billing. The Studio's Chroma presets attach consentId: consent_operator_isis_dev_studio (operator's own consent for a typed prompt); curl submissions must pass parameters.consentId themselves.

  • Images are watermarked (C.12.13, was: unwatermarked by policy). The worker-comfyui path now embeds a provenance payload — isis:v1;job=…;consent=…;wf=…;model=…;produced=… — in the blue channel's low bit of every inline PNG output, before the bytes are persisted, and records sha256(payload) as the bundle's watermarkHash. The payload is framed with a CRC and repeated (512 times over a 1024² image), so damage to part of the picture still yields it back. Measured on the C.11 fixture: +1.9% file size versus ComfyUI's own encoder, +0.5 s, every sample moved by at most 1.

    What it does not do, on purpose: spatial LSB survives a lossless re-save and the storage path, but not JPEG/WebP re-compression, resizing, cropping or rotation. It is a provenance marker for the artifact Isis stores and serves, not a forensic watermark against a determined adversary; a transform-domain (DWT-DCT) version is separate, larger work.

    ISIS_PROVENANCE_ALLOW_UNWATERMARKED_IMAGES survives with a narrower job and should be off: it decides only what happens to an output the stage could not mark. Off refuses that job's result at completion; on keeps it with watermark: { applied: false, reason } recorded. A job with no still image to mark (a clip, a mask-only flow) is not covered by it either way.

  • Runtime, settled 2026-09-11 21:39Z. The endpoint is 6l1f3tku7z2zoy (isis-comfyui-chroma), built by RunPod's git integration from branch runpod-comfyui-slim. Its active release d677f914 (commit 105ad3cf) works — it creates containers, boots ComfyUI and renders. The interim stock-image endpoint ac988l8is5axkf that produced the first image was deleted and is no longer needed; template zxojew9tyo survives as a fallback only.

  • If the endpoint stops placing workers, re-save it in the console. After a GraphQL saveEndpoint call this endpoint went dead: zero workers in every state with jobs queued, and a REST PATCH {"workersMax":1} did not revive it. Manage → Edit endpoint → Save Endpoint, re-applying the same configuration, fixed it immediately. Never call GraphQL saveEndpoint on a git endpoint; if one is already wedged, the console form is the cure, not a rebuild.

  • "Low Supply" is not a blocker. The edit dialog labels EU-RO-1's selected pools Low Supply / Unavailable, yet a worker was placed within ~40 s. Do not conclude a capacity problem from that label alone, and do not propose migrating datacenters over it — a network volume cannot move, so migrating costs a second 500 GB volume plus re-seeding ~70 GB.

  • Cheapest possible liveness probe (costs nothing if no worker boots, ~$0.03 if one does): submit a model-free graph and purge afterwards.

    bash
    curl -s -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
      -d '{"input":{"workflow":{
        "3":{"class_type":"EmptyImage","inputs":{"width":64,"height":64,"batch_size":1,"color":8421504}},
        "4":{"class_type":"SaveImage","inputs":{"images":["3",0],"filename_prefix":"smoke"}}}}}' \
      "https://api.runpod.ai/v2/$RUNPOD_COMFYUI_ENDPOINT_ID/run"
    # then: POST https://api.runpod.ai/v2/<ep>/purge-queue
    
  • GPU pools, not GPU names. EU-RO-1 (the volume's datacenter) had no free 4090 and does not offer A5000/3090; both endpoints use pools ADA_24,AMPERE_24,BLACKWELL_32 set through GraphQL saveEndpoint (the REST PATCH silently ignores gpuTypeIds). Renders land on RTX PRO 4500 Blackwell.

  • Measured cost. Cold run: 53 s queue/boot + 150 s GPU = $0.083 at the 4090 rate; the model load from the volume (T5 ~40 s, Chroma1-HD ~65 s) is most of the GPU time. Ledger rate stays 0.000306 $/s until RunPod's invoice for the Blackwell pool is compared (C.11.13).

Restarting the API mid-job is survivable now (C.12.16)#

A job that was in flight when the API process died used to be lost outright: the inline runner claims only pending/queued, so the row sat at running forever while RunPod finished the render and billed for it (job 7d4280f9, 148.98 s, gone). Two things changed:

  • The RunPod job id is written to job.metadata the instant RunPod accepts the submission, before the first status poll — providerJobId, providerEndpointId, providerDispatchedAt.
  • The runner sweeps once at startup, before it claims anything. A running job whose owning process is provably gone (inline-<pid> and the pid is dead, or the pinned ISIS_INLINE_JOB_RUNNER_ID is this process's own) is put back in the queue with its dispatch record intact. The next claim re-attaches to the existing RunPod job rather than submitting a new one, so nothing is rendered or billed twice, and the cost ledger is idempotent per job id anyway.

What it does NOT do, on purpose: a job whose worker id cannot be proved dead is left exactly where it is, and logged as left with the reason. Two processes running the same job would double-bill it, which is worse than a stuck row. A job that has no providerJobId (dispatched before this change, or the write failed) is failed with ORPHANED_WITHOUT_PROVIDER_JOB_ID rather than left lying about being running — nothing can say which render was its.

Watch the startup log for Re-queued an orphaned job for re-attachment and Re-attaching to an already-dispatched RunPod job instead of submitting. Note the CLAUDE.md rule still stands for its own reasons: do not edit app source while a run is in flight, because tsx watch recompiles under it.

2e. Cold boot on this endpoint got much slower when the video packs landed#

Measured 2026-09-11, before the video packs: a fresh EU-RO-1 host pulled the 14.7 GB worker image in ~4 minutes (job d1e87d06).

Measured 2026-09-12, after V.01.01 added six video node packs and their dependencies (cupy, triton, pycocotools, timm, the controlnet_aux annotators): a fresh host was still initializing after 15 minutes, and runpod-catalog-run.mjs gave up in IN_QUEUE without the job ever starting. Nothing was billed — delayTime is not GPU time — but nothing ran either.

What this means in practice:

  • A cold start here is minutes, plural, and possibly many. Plan a session around one boot: submit the first job, let the worker come up, then run everything else back to back while it is warm (idleTimeout is 10 s, so "warm" means within seconds of the previous completion, not minutes).
  • initializing is not the same as wedged. The endpoint fault C.12.12 recorded shows zero workers in every state with jobs queued. One worker initializing means placement worked and the host is pulling the image. Do not re-save the endpoint, do not rebuild, and do not resubmit — the queued jobs run when the boot finishes.
  • A queued job survives the client giving up. The runner's 15-minute cap abandons the wait, not the job. Read it back later with GET /v2/<endpoint>/status/<jobId>; do not submit a second one, which just queues behind the first.

2d. Video extension (V.10.01)#

Nothing here is needed for the image stack. Turn it on as a block — a half-configured video endpoint is the dangerous state, not an unconfigured one, which is why dev-stack-preflight.mjs skips every video check while RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID is empty and binds all of them the moment it is set.

  • RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID — the second endpoint (V.03.04 prints it on creation; scripts/isis/runpod-provision.mjs endpoint list recovers it later). Every motion-category workflow routes here. There is no fallback to the image endpoint — a motion submit with this unset fails loud (V.06.03). Both endpoints run the same image off the same volume; the difference that matters is the card, and falling back would send a 28 GB expert to a 24 GB pool and bill the OOM (comfyui.service.ts:967).

  • ISIS_COST_RATE_RUNPOD_VIDEO_USD_PER_SECOND=0.001647 — H200 141 GB flex ($5.93/h), not the H100's 0.001331. The endpoint's gpuIds list includes H200, and a list means "any of these": the ledger cannot know which card ran, so it has to bill the dearest or it under-counts by 24% every time an H200 is assigned. (It carried 0.001331 until 2026-09-12 because the preflight's price table had no H200 row and reported it UNPRICED rather than guessing — correct behaviour that still left the rate wrong; the table now prices all seven enabled cards from §2f's measured tiers.) Copying the IMAGE rate here is the other mistake to fear: an H200 second costs ~3.8x a 5090 second, so the ledger would under-count by that factor and the budget would never fire. The preflight asserts this against the comfyui-video entry's gpuIds in infra/runpod/endpoints/desired-state.json and fails if the configured rate is below the priciest GPU that endpoint may be assigned (0.000486 if V.11.06 moves Wan to L40S).

  • ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6 — a second ledger with its own Redis key (:video), separate from the image one. A single video job can cost more than the entire image day budget: a 5-second 720p Wan 2.2 clip is a few minutes of H100 time, and the image budget is $3/day total. The preflight caps this at 50 for the same reason.

  • Volume S3 (RUNPOD_S3_ACCESS_KEY, RUNPOD_S3_SECRET_KEY, RUNPOD_VOLUME_ID=5bst2pgw7y (the model cache; ax0oj250j3 was deleted 2026-09-13), RUNPOD_S3_ENDPOINT=https://s3api-eu-ro-1.runpod.io, and BUCKET_ENDPOINT_URL for the worker side). Inputs and outputs above the inline ceiling ride the volume rather than the request body. The preflight checks the two credentials by LENGTH only and never prints them, and refuses a BUCKET_ENDPOINT_URL whose host disagrees with RUNPOD_S3_ENDPOINT or whose path does not name the volume — pointing the worker's bucket anywhere else writes outputs where nothing reads them.

  • Retention sweep (ISIS_VOLUME_RETENTION_SWEEP_ENABLED, ISIS_VOLUME_RETENTION_DRY_RUN, ISIS_VOLUME_RETENTION_DAYS, ISIS_VOLUME_RETENTION_SWEEP_INTERVAL_MS). The API can delete inputs/<jobId>/ and outputs/<jobId>/ objects past an age, so the volume does not get grown to store last month's test clips. It is off unless enabled and a dry run even then — set ISIS_VOLUME_RETENTION_DRY_RUN=false to actually delete, and read the dry-run log line first; it is the same line the real run prints. models/ is not sweepable at any age and the refusal is in the client, not the config. By hand: node scripts/isis/runpod-volume-s3-put.mjs sweep --days 7 (add --yes to delete).

ffprobe is a convenience here, not a dependency. The API reads a clip's real duration, fps, frame count and dimensions out of the MP4 boxes itself (apps/isis/generation-api/src/services/mp4-probe.ts), precisely because ffprobe is silently absent on a slim image and a stack that depends on it fails in a way nobody notices. Install it anyway for cross-checking a suspicious clip by hand:

bash
ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height,nb_frames,avg_frame_rate \
  -show_entries format=duration -of default=nw=1 out.mp4
# no output at all -> not installed; `brew install ffmpeg` (macOS),
# `apt-get install -y ffmpeg` (Linux dev server). Nothing in the API needs it.

Two honest caveats when comparing its numbers to the API's: mp4-probe reports the AVERAGE fps over the track (frames / track duration), so a clip truncated by -shortest muxing reads lower than its nominal rate — measured at 12.8 vs 16 on such a file, while every untruncated clip (5/9/17/33/81 frames at 16 fps) agrees exactly. And WebM is a different container (EBML): the parser returns null for it rather than guessing, so an alpha .webm keeps whatever the graph declared.

2f. Stopping the video endpoint, and what happens when there is no GPU (V.03.05)#

Endpoint 2lezocymx2rgot (isis-comfyui-video), EU-RO-1, workersMin 0 / workersMax 1, 900 s execution cap.

Kill switch — workersMax to 0. Either the dashboard's per-endpoint button (runpod-kill-switch.service.ts, target video) or:

bash
node scripts/isis/runpod-provision.mjs endpoint update 2lezocymx2rgot --workers-max 0 --yes
node scripts/isis/runpod-endpoint-drift-check.mjs --key comfyui-video   # expect DRIFT on workersMax

A bare runpod-endpoint-drift-check.mjs with no --key now checks every endpoint in the environment and says how many it checked; --key narrows it to one.

A paused endpoint queues NOTHING — it refuses the submit. Measured against the live video endpoint at workersMax: 0 (V.11.04, 2026-09-12), /run answers:

json
{
  "status": 409,
  "title": "Conflict",
  "code": "ENDPOINT_PAUSED",
  "detail": "Endpoint is paused (max_workers=0). Set max_workers > 0 to accept work."
}

/health stayed at inQueue: 0 with no worker in any state. So the switch is stronger than "nothing new starts": there is no backlog waiting to stampede when the ceiling goes back up, and nothing to cancel. The API surfaces it as RUNCOMFY_ENDPOINT_PAUSED with the message "The RunPod endpoint is stopped (workersMax is 0) and is accepting no work. Raise workersMax in the RunPod console to resume.", and does not retry it — a paused endpoint stays paused until a human raises it, so every retry would be another 409. (It arrived as "Unexpected RunPod provider error." until this was fixed.)

Raising it again is deliberate and manual. Two more things measured 2026-09-12:

  • /run keeps answering 409 ENDPOINT_PAUSED for ~30 s after you raise workersMax again. REST reports the new value immediately. That lag is propagation, not the wedged-endpoint fault — wait half a minute and retry before diagnosing anything.
  • A job that DID reach the queue before the pause outlives the client that submitted it. runpod-catalog-run.mjs gives up after 15 minutes; the job does not. Read it back with GET /v2/<endpoint>/status/<jobId> rather than submitting a second one, which only queues behind the first.

No GPU available: the job waits, and then the API gives up. It does NOT fall back to a smaller card. That refusal is deliberate — a 28.6 GB fp16 Wan expert that OOMs has already been billed for the load, so a "helpful" fallback to a 48 GB card converts a wait into a charge for nothing. The endpoint is pinned to cards that all clear the 70.53 GB peak (three RTX PRO 6000 Blackwell variants, three H100 variants, H200) and nothing smaller is enabled.

While the job sits IN_QUEUE nothing is billed — RunPod's delayTime is not GPU time — so the cost of waiting is wall clock only. What ends the wait is the queue/boot allowance from C.12.17 (ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS, default 3 600 000 — one hour), after which the API cancels the job as retryable because nothing was spent.

Every queue-side deadline is one hour as of 2026-09-12, and they are a different thing from the execution caps. Queue wait is unbilled, so a short deadline buys nothing and throws away work that was about to run:

Deadline Value Where
API queue/boot allowance, per attempt 1 h ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS
Matrix runner give-up (cold start only) 1 h motion-live-proof.mjs
Single catalog run 1 h runpod-catalog-run.mjs
Orphan collector 1 h motion-proof-collect.mjs --deadline-min
Execution cap, image 5 min ENDPOINT_EXECUTION_TIMEOUT_MS.image
Execution cap, video 15 min ENDPOINT_EXECUTION_TIMEOUT_MS.video

The last two bound BILLED GPU time and are the cost guard. Do not raise them to match the queue ones.

Supply in EU-RO-1 is the real constraint, and it is not what the console shows you first. Measured 2026-09-12, the same three tiers before and after attaching the network volume:

Tier Before volume (global) After volume (EU-RO-1)
96 GB, $3.49/h High Supply Low Supply
80 GB, $4.79/h High Supply Unavailable
141 GB, $5.93/h Medium Supply Unavailable

So the three-deep fallback is, in practice, a chain of one. All three stay enabled because "Unavailable" is a point-in-time reading rather than a property, but plan on 96 GB or a wait. Never size a RunPod endpoint from the labels shown before the volume is attached — they describe datacenters you will not be deployed in.

Re-measured 18:20Z the same day, with every tier the console offers, because V.11.01 could not get a card at all (three attempts, throttled, $0.00 billed, and the console saying "There's no free GPU capacity for your current selection"):

Tier Enabled Supply $/hr
96 GB Pro 1st Low Supply 3.49
80 GB Pro (H100) 2nd Unavailable 4.79
141 GB (H200) 3rd Unavailable 5.93
80 GB (A100) 4th Unavailable 2.72
16 GB no Medium Supply 0.58
24 GB Pro no Low Supply 1.10
24 / 32 / 48 GB no Unavailable ≤ 1.75
180 GB Pro no Unavailable 8.64
288 GB Pro no Unavailable 10.65

Every tier that can hold a 70.53 GB Wan expert is Unavailable except the 96 GB Pro one, which is Low Supply. Adding the two A100 80 GB variants (done the same day, endpoint version 2) did not help: that tier is dry too. There is no card selection that fixes this — the only ≥80 GB tier with any supply is the one that was already first in priority. If a clip has to run, the region is the only lever left; see §2g. Note also that a worker can reach ready and be taken back before it picks up a job (observed 18:14:40Z → 18:15:00Z), so a ready reading is not yet capacity.

Two practical consequences when supply IS found:

  • Nothing is billed while a job waits or a worker is throttled — the console's billing panel read $0.00 across all three attempts. Waiting costs wall clock only, and the C.12.17 queue-phase timeout retries it for free.
  • The 10 s idleTimeout is dangerous in a low-supply region: lose the worker between two clips of a warm session and the next one rejoins the same queue with no guarantee of a card. Raise it for the duration of a multi-clip session (V.11.02) rather than paying for a second placement gamble.

2g-migration. Moving to US-CA-2 (in progress 2026-09-12)#

Closed 2026-09-12 23:58Z — the stack stays in EU-RO-1. User decision after the §2g-full-survey below: no region has proven supply of both card tiers, US-CO-1 is H200-only, and US-CA-2 never placed a worker. Everything outside EU-RO-1 was deleted in one pass (endpoints eron7rmrmcg0rq, ajcpag53ed052j, ay4gcv868ein1u; their templates plus the seven probe templates; volumes sq5230axbc US-CA-2 and fj3f77pca3 US-CO-1). What remains is exactly the pre-migration state: volume ax0oj250j3, endpoints 2lezocymx2rgot and 6jye7vngg4i685. The recipe below is kept because it is correct for any future move — read the survey and the region-hunt rules first.

Approved after the probe below. Nothing in EU-RO-1 is touched until a clip has rendered in US-CA-2 — the 500 GB volume there is the only copy of 358 GB that took hours to fetch, and a half-finished migration that deleted it would be unrecoverable.

bash
# 1. Volume in the new region. A volume PINS the datacenter, which is also why
#    the existing endpoints cannot follow it — new ones are required.
curl -s -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
  -d '{"name":"isis-chroma-models-us","size":500,"dataCenterId":"US-CA-2"}' \
  https://rest.runpod.io/v1/networkvolumes          # -> sq5230axbc

# 2. Loader pod. CPU-only ($0.06/h); V.02.04 forbids a GPU pod for this.
node scripts/isis/runpod-provision.mjs pod create-loader \
  --volume sq5230axbc --dc US-CA-2 --yes            # -> mf35u5m5k0c8xi

# 3. Watch it. The .part suffix is the loader resuming with HTTP Range.
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-2.runpod.io
node scripts/isis/runpod-provision.mjs volume ls sq5230axbc --dc US-CA-2

# 4. The verdict is the loader's own, not a size check: it verifies every
#    sha256 and exits 0 only if all 42 rows are "ok".
node scripts/isis/runpod-provision.mjs volume cat sq5230axbc .oshun/manifest-result.json --dc US-CA-2

# 5. Delete the pod the moment the pull is done — it bills while it lives.
node scripts/isis/runpod-provision.mjs pod delete mf35u5m5k0c8xi --yes

A CPU pod cannot run the worker image. POST /pods with computeType: CPU rejects containerDiskInGb above 20, and both live templates for the video image declare 30. A CPU pod created at 20 GB sat 13 minutes with runtime: null (pull never finished) before being deleted. Anything that needs the image's own filesystem — the build-time object_info.json in /opt/oshun/build-manifest/, for one — has to run as a serverless GPU worker on the same volume.

Set workersStandby: 0 explicitly on the new endpoints. Creating a serverless endpoint through the REST API defaults it to 3 (and workersMax to 3), and a standby worker starts billing with no job submitted — see the probe note below. The two EU endpoints carry workersStandby: 1 and are not billing for it only because EU-RO-1 cannot place a worker at all.

Order of the cutover, deliberately conservative: pull → verify all 42 sha256s → create both endpoints on the new volume → render one clip → only then delete the EU volume and its endpoints. The EU stack keeps serving throughout.

Everywhere the region is pinned. Found by grep rather than memory, because a missed one fails as "no GPU available" or a silently wrong provenance record rather than as an error:

File / setting Now At cutover
infra/runpod/volumes/isis-chroma-models.manifest.json → volume.dataCenterId EU-RO-1 US-CA-2
…same file → volume.name isis-chroma-models isis-chroma-models-us
infra/runpod/endpoints/desired-state.json → both dataCenterId EU-RO-1 US-CA-2
…both networkVolumeId ax0oj250j3 sq5230axbc
…both endpointId EU ids the new ids
…both workersStandby not declared (RunPod default) 0, explicitly
Env recipe RUNPOD_REGION EU-RO-1 US-CA-2 — it is the provenance bundle's region, so a stale value mislabels every render
Env recipe RUNPOD_S3_ENDPOINT https://s3api-eu-ro-1.runpod.io https://s3api-us-ca-2.runpod.io
Env recipe RUNPOD_VOLUME_ID ax0oj250j3 sq5230axbc
Env recipe both endpoint ids EU ids the new ids

The manifest is shared between the two volumes while both exist, so its volume block is changed at cutover and not before — editing it early would make the lint describe a volume that is still serving.

RUNPOD_REGIONS in libs/isis/runpod-surface/src/endpoint-registry.ts is an AWS-style list (us-east-1, …) unrelated to RunPod's datacenter ids and needs no change; RUNPOD_REGION is a free string read only for the provenance region.

The S3 listing returns intermittent HTTP 524. Two tries in three failed while the loader was writing. It is Cloudflare timing out on the listing, NOT an empty volume — retry before believing a zero, and never report an error as 0 GB.

2g-region-hunt. US-CO-1 grants cards; US-CA-2 and EU-RO-1 do not (2026-09-12 23:2xZ)#

How to test a region, since the obvious routes do not work. The console's supply labels need a volume to pin the datacenter and are far too slow to read a dozen times. GraphQL introspection is disabled. lowestPrice(input:{dataCenterId}) returns almost nothing per-DC and quotes B200 at the same $5.98 everywhere — it is the POD market, not serverless. So test placement, which is the ground truth anyway and is pure API:

  1. Create a 10 GB volume in the datacenter (a volume is what pins it).
  2. Create a throwaway endpoint on it with the cards you actually want, and a tiny public image so nothing waits on a pull.
  3. Read /v2/<id>/health and count ready + idle + running.
  4. workersMax: 0, then delete endpoint → then volume (a volume attached to a live endpoint returns 500).

initializing is NOT evidence of a card. It is equally what a worker shows while WAITING for one — the Workers tab spells it out: "Waiting for GPU to become available…". Only ready/idle/running proves a card was granted. Counting initializing as success will tell you a starved region is healthy.

Measured, same eight large cards offered to all four:

Datacenter Card granted? Evidence
US-CO-1 yes, repeatedly placed=2 at 23:21, 23:22 and 23:23
US-GA-2 once, not repeatable ready/idle on the first read only
US-CA-2 no 1 h+ queued; the V.11.02 verification run gave up having never started, $0
EU-RO-1 no initializing all session, never placed

The account caps at 5 max-workers across all endpoints, so probes have to be run in batches and torn down between them.

One placement is not evidence. US-CA-2 was migrated to on the strength of a single worker appearing within a minute at 20:02Z, and it has not granted a card since — the 358.30 GB now sits in a region that cannot serve it. US-CO-1 is recorded here only because it held across three consecutive readings. Apply that bar before moving anything again.

2g-full-survey. Every storage-capable datacenter, one form read (2026-09-12 23:50–23:53Z)#

The question that should have been asked before any volume was created: which regions have the card tiers BOTH endpoints need? Read from the console's deploy form with exactly one datacenter selected and no volume, for all seventeen datacenters GraphQL marks storageSupport: true — a volume pins the region, so nowhere else can host the stack. Raw labels and prices: docs/agents/evidence/isis-chroma-runpod/region-supply-survey-2026-09-12T2350Z.json.

Image needs 24 / 24 PRO / 32 PRO ($0.69 / $1.10 / $1.58 per hour). Video needs 80 PRO (H100, $4.79), 96 PRO (RTX PRO 6000, $3.49), 141 (H200, $5.93) or 180 PRO (B200, $8.64). — is Unavailable.

Region 24 GB 24 PRO 32 PRO 80 PRO 96 PRO 141 GB 180 PRO
EU-RO-1 Low Medium Low — Low — —
US-CA-2 Low Low — Low Low Low Low
US-CO-1 — — — — — Low —
US-GA-2 Low — — Low — Low —
US-NC-2 — — — — Low — Low
EUR-IS-1 Low — — — Low — —
CA-MTL-3 — Low — — Low Low —
AP-JP-1 — — — Low — Low —
EU-FR-1 — — — Low — Low —
EUR-IS-3 — — — Low — — —
EUR-NO-1 — Low Low — — — —
EUR-NO-2 — — — Low — — —
US-IL-1 Low High — — — — —
US-MO-2 Low — — — — — —
US-TX-3 — Low — — — — —
CA-MTL-4 — — — — — — —
EU-NL-1 — — — — — — —

What it settles.

  • US-CO-1 is an H200-only datacenter. Three independent readings agree: this form, the earlier console table in §2f, and GraphQL dataCenters.gpuAvailability (which lists exactly one card there). The workers the region hunt saw placed can only have been H200s. So US-CO-1 is the most expensive place to run video short of B200 ($5.93/h — 1.7× the 96 GB PRO rate, 2.2× the A100 the EU clip actually billed at), and it has nothing at all for the image endpoint: the chroma stack could not follow the volume there. The region hunt measured "does a worker get placed" and never asked "on what card" — that is the gap this table closes.
  • No region has proven supply of both tiers. Five carry labels for both (EU-RO-1, US-CA-2, US-GA-2, EUR-IS-1, CA-MTL-3); only EU-RO-1 has ever rendered anything for us, and US-CA-2 — the widest label coverage in the table — never placed a worker in three hours. A Low label does not predict placement in either direction: US-CA-2 had four Low tiers and placed nothing; US-CO-1 had one and placed repeatedly. Placement tests (§2g-region-hunt) remain the only ground truth, and they must now also record the card granted.
  • EU-RO-1 is the best image region on the board — Medium on 24 PRO, the only Low-or-better 32 PRO besides EUR-NO-1, and every image job has rendered there. The image endpoint has no reason to move.
  • Candidates for video that are untested: US-NC-2, CA-MTL-3 and EUR-IS-1 list 96 GB PRO. Each needs the placement test with the video endpoint's exact card list before a byte is copied — the US-CA-2 migration is what happens when that step is skipped.

Three volumes are billing right now (EU-RO-1 ax0oj250j3, US-CA-2 sq5230axbc, US-CO-1 fj3f77pca3; $35/month each at 500 GB). US-CA-2 has never served a job and US-CO-1 is empty; at most one of the two should survive the decision.

US-CO-1 has no pod market and no S3 API. POST /pods rejects it (the REST dataCenterIds enum predates the region; without the field, every CPU flavour answers "no instances available"), and s3api-us-co-1.runpod.io does not resolve. The only way to write to a volume there is a serverless worker in the region — a CPU-serverless loader endpoint (ay4gcv868ein1u, template e2gcmznv3p, python:3.12-slim running the C.02.06 loader behind a tiny runpod handler that returns manifest-result.json as job output) was created for that and parked at workersMax: 0 pending this decision.

2h. The object-info snapshot comes out of the build log#

infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json is the worker image's own /object_info, reduced to what the graph validator reads, and it is refreshed like this — no GPU, no pod, no S3:

bash
git push origin HEAD:runpod-comfyui-slim          # RunPod rebuilds; smoke emits OBJINFO lines last
node scripts/isis/runpod-build-log.mjs --latest --endpoint 2lezocymx2rgot   # wait for COMPLETED
node scripts/isis/runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot        # verifies chunks + sha256, writes the file
npx vitest run --root libs/isis/workflows graph-validator                    # every golden against it

Why the log: getBuildLogs takes only buildId and startTime and returns the newest 500 lines, so the smoke prints the snapshot as the very last thing (75 lines of 1600 chars for 1071 classes). A CPU pod cannot hold the image (containerDiskInGb ≤ 20, the image needs 30), US-CO-1 has no S3 API, and a GPU worker in EU-RO-1 waited over an hour without a card. The root .dockerignore excludes infra; the manifest is re-included by name so the smoke can shadow the volume — a new file under infra/ that the build needs must be re-included the same way.

2g-supply. Both regions fluctuate; neither is reliably warm#

The honest comparison, each read off the endpoint's own GPU configuration list:

Tier EU-RO-1 @ 18:20Z US-CA-2 @ 22:55Z
96 GB Pro, $3.49/h Low Supply Unavailable
80 GB Pro H100, $4.79/h Unavailable Low Supply
141 GB H200, $5.93/h Unavailable Low Supply
80 GB A100, $2.72/h Unavailable Unavailable
180 GB B200, $8.64/h Unavailable Low Supply

US-CA-2 has two of the enabled large tiers at Low Supply where EU-RO-1 had one, plus B200 — which is why the move is defensible. But at 22:55Z US-CA-2 was ALSO answering "There's no free GPU capacity for your current selection" with two jobs queued. Neither region is reliably warm, and a single placement is not evidence that one is. The probe that triggered this migration got a card in US-CA-2 in under a minute at 20:02Z; the same selection could not get one three hours later. Read the per-tier table before concluding anything about a region.

Initializing does not mean "pulling the image". It is also what a worker shows while it waits for a card it has not been given. The Workers tab says which — "Waiting for GPU to become available…" — and the Logs tab stays empty ("logs will appear here once they're ready"). Do not infer a pull from the status string.

B200 is enabled on the US video endpoint (2026-09-12) because it was the only large tier showing supply there and it is the one plausible shot at wan22-animate, which died at the floor of the caps and may never have had a card big enough. It costs $8.64/h, so it is now the dearest card either endpoint can be assigned and the video ledger rate rises to $0.0024/s — every video estimate goes up about 46%, which makes the budget gate more conservative, not less.

2g-billing. A first-time image pull in a new region is NOT billed#

Read off the endpoint's own Billing panel while a worker had been initializing for 27 minutes on its first-ever pull in US-CA-2:

No charges in the past 24 hours · Spend rate $0.00000 / second · Total billed $0.00

So the picture is now complete and consistent with V.11.01's account evidence: RunPod bills execution. Not queue wait, not worker boot, not image pull. That is why our ledger charging delayTime overstated a clip by 4x, and it is why a cold region is worth waiting out rather than abandoning — a 27-minute first pull costs nothing but wall clock.

Check the panel rather than reasoning about it. The endpoint page shows Spend rate and Total billed live, and it answered in one page-load a question that is otherwise unanswerable from the API.

2g-probe. US-CA-2 has capacity EU-RO-1 does not (measured 2026-09-12)#

The migration question is no longer speculative. A 10 GB volume in US-CA-2 with a throwaway endpoint carrying the identical nine cards as the video endpoint was handed a worker inside a minute, with no jobs queued at all, while EU-RO-1 at the same instant sat throttled with two jobs waiting. Same card ids, opposite outcome: the constraint is the REGION.

Caveats, because one reading is one reading: the probe ran a tiny runpod/base image, so it measured card availability and not boot time — a 288 GB volume and a multi-GB image still have to be pulled wherever we go. US datacenters that support network volumes: US-CA-2, US-CO-1, US-GA-2, US-IL-1, US-MO-2, US-NC-2, US-TX-3.

Two traps the probe walked into, so you do not have to.

  1. Creating a serverless endpoint through the REST API starts a worker immediately, with no job submitted. RunPod defaults workersStandby to 1 (3 when workersMax is left unset), and workersStandby cannot be set at all through REST — it is rejected as an unknown key on both create and patch. So every endpoint you create begins billing at once.

    PATCH {"workersMax": 0} does stop it, within about 40 seconds. An earlier note here said it did not and that only deleting the endpoint worked; that was wrong — the probe was deleted ~20 s after the patch, before the worker had wound down. Measured on isis-comfyui-video-us: initializing at t+20 s, no workers at t+40 s.

    So the recipe is: create → immediately workersMax: 0 → raise it to 1 only when you are ready to submit. That costs well under a minute of one worker instead of however long you take to notice.

  2. Both real endpoints carry workersStandby: 1. They are not billing for it today only because EU-RO-1 cannot place a worker at all. That is starvation suppressing a standing cost, not a safe configuration — re-check it before moving to a region that can actually serve us.

2g. The image endpoint was rebuilt, and the region option if GPUs run out#

Endpoint 6jye7vngg4i685 (was 6l1f3tku7z2zoy, deleted 2026-09-12), template vfl3iu1dr6, container disk 30 GB.

Why it was rebuilt: container disk. The old endpoint's template allocated 20 GB. When V.01.01 added the six video node packs and their dependencies (cupy, triton, timm, pycocotools, the controlnet_aux annotators) the image outgrew it, and from Release #10 onward no worker ever started — placement succeeded, the container was never created, and the worker sat initializing indefinitely with completely empty logs. Raising the disk on the old endpoint did not recover it; a rebuild did, and the first job on the new one ran normally.

If a worker ever hangs in initializing with no log output again, check the container disk against the image size first. It is the one field neither API exposes for a git-integration template — GET /v1/templates/<id> 404s and GraphQL podTemplates omits it — so it is console-only, under Manage → Edit endpoint.

Region is NOT the usual cause, but it is the fallback. Serverless supply per region, measured in the console's endpoint-create form on 2026-09-12 with exactly one datacenter selected (labels are point-in-time and moved measurably within the same session):

Region 24 GB 24 PRO 32 PRO 48 PRO 80 PRO 96 PRO 141 GB
EU-RO-1 (current) Low Low Low — — Low —
US-CA-2 — Low — — Low Low Low
US-IL-1 Low High — Low — — —
US-GA-2 Low — — — Low — Low
US-MO-2 Low — — — — Low —
US-TX-3 — Low — Low — — —
US-NC-2 — — — — — Low —
US-CO-1 — — — — — — Low

Read it this way: EU-RO-1 is the only region with 32 GB PRO, which matters because Chroma1-HD (17.8 GB) plus T5-XXL (9.8 GB) is a tight fit in 24 GB. US-CA-2 is the only region covering both tiers — image and all three video sizes — so it is the migration target if one is ever needed. US-IL-1 has the only High reading anywhere (24 GB PRO) but nothing for video.

Trigger for migrating (user decision 2026-09-12): only if no GPU available / sustained throttled starts recurring. It is not worth doing pre-emptively. The move is: create a 500 GB volume in the target region, run the C.02.06 loader against the same manifest (~$0.06 of CPU pod, ~1 hour for 358 GB), verify all 42 sha256s, rebuild both endpoints there, and only then delete the EU-RO-1 volume. Keep both volumes until the new one verifies — the overlap is about $0.05/h.

Two API traps if you do migrate. NVIDIA RTX PRO 4500 Blackwell — the card that ran every successful C.11 job — is not accepted by the serverless REST API, and PATCHing it sets gpuTypeIds to null rather than erroring, leaving the endpoint with no GPUs at all. It is only reachable as the pool id BLACKWELL_32, which REST rejects, and the console's "32 GB PRO" tier maps to ADA_32_PRO instead. Also, REST reports a stale gpuTypeIds for an endpoint whose GPUs were changed through the pool API — trust the console.

2b. Dev credential#

Provider routes (/api/v1/jobs/runpod/*, /ready detail) need a bearer token with an operator role; the API has no API-key validator wired. Add a throwaway JWT_SECRET (≥ 32 chars, e.g. openssl rand -hex 32) to the env recipe, then:

bash
TOKEN=$(node scripts/isis/mint-dev-jwt.mjs --hours 12)   # HS256, admin role, project_isis_dev
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:4010/api/v1/jobs/runpod/cost/today

Paste the same token into the dashboard's Login page (bearer token field).

3. Preflight (refuses to continue on any incoherence)#

bash
node scripts/isis/dev-stack-preflight.mjs
node scripts/isis/validate-chroma-endpoint-config.mjs --require-image

4. Generation API (port 4010)#

bash
memory_pressure | tail -1
( cd apps/isis/generation-api && pnpm dev ) > /tmp/isis-gen-api.log 2>&1 &
echo $! >> /tmp/claude-<session>.pids
until curl -sf http://127.0.0.1:4010/health >/dev/null; do sleep 2; done
curl -s http://127.0.0.1:4010/health | head -c 400
curl -s http://127.0.0.1:4010/api/v1/jobs/runpod/cost/today

Log lines to expect: Inline job runner started, Workflow catalog bootstrap complete … includeChromaCatalog: true, RunPod adapter initialized.

tsx servers do not pick up source edits — restart after any change.

5. Dashboard (Vite, port 5180)#

bash
( cd apps/isis/web && VITE_ISIS_GENERATION_API_URL=http://127.0.0.1:4010 pnpm dev ) \
  > /tmp/isis-web.log 2>&1 &
echo $! >> /tmp/claude-<session>.pids

Open http://127.0.0.1:5180 → Login accepts a pasted bearer token or API key; with NODE_ENV=development the API allows anonymous access.

The Workflows page is backed by the separate workflow-registry service (user-authored workflows, port 3003), which does not know the catalog, so the Chroma flows are submitted from Job Submission Studio: type text-to-image (or image-to-image / inpainting), prompt, and in Extra parameters (JSON):

json
{
  "workflowId": "chroma-txt2img",
  "checkpoint": "chroma1-hd",
  "aspect": "square_1024"
}

Every catalog input travels under its own name (checkpoint, aspect, cfg, steps, seed, denoise, model, text_prompt, grow_mask_by, confidence_threshold); image inputs (source_image, mask_image) take a data:image/png;base64,… string. The Jobs page shows the run (claimed by inline-<pid>) and, once completed, the output thumbnail with seed, checkpoint, measured cost and execution time; the RunPod page shows spend.

5b. Worker outputs to Hetzner Object Storage (C.12.06 / V.12.05, decided 2026-09-13)#

The stock worker's uploader names its bucket after the month and cannot sign for a non-AWS host, so the image carries an oshun_uploader.py (imported at start-up through an oshun_uploader.pth) that replaces rp_upload.upload_image whenever BUCKET_ENDPOINT_URL is set (see docker/runpod/comfyui-slim/oshun_uploader.py). Outputs then land in Hetzner Object Storage and come back to the API as presigned URLs, which the API fetches into its own output store — so the browser still sees one kind of URL.

Set these on BOTH endpoints in the RunPod console (Manage → Edit endpoint → Environment variables; a git-integration template's env cannot be edited through either API). The two secrets come from ~/Desktop/workspace/drawer/env-master.env (HETZNER_APP_DEV_S3_ACCESS_KEY, HETZNER_APP_DEV_S3_SECRET_KEY); copy the values, never commit them.

Key Value
BUCKET_ENDPOINT_URL https://hel1.your-objectstorage.com
BUCKET_ACCESS_KEY_ID the Hetzner access key
BUCKET_SECRET_ACCESS_KEY the Hetzner secret key
BUCKET_NAME app-dev-server
BUCKET_KEY_PREFIX isis/outputs
BUCKET_REGION hel1
COMFY_LOG_LEVEL INFO (the DEBUG default floods the log)
COMFY_POLLING_MAX_RETRIES 1500 (image) / 3600 (video)
COMFY_POLLING_INTERVAL_MS 250

Objects are keyed isis/outputs/<jobId>/<8 hex><ext>. The API side needs ISIS_WORKER_OUTPUT_FETCH_HOSTS=hel1.your-objectstorage.com (§2); any other host in a worker's s3_url is refused with the reason recorded on the output. The worker log prints oshun uploader: rp_upload.upload_image -> bucket … at start when the patch is active, which is the check that the env reached it.

5c. Model library and cache (option B, tracker L.00–L.07, decided 2026-09-13)#

Where weights live. The source of truth is the Hetzner library: bucket app-dev-server on https://hel1.your-objectstorage.com, prefix isis/models/, key = prefix + the manifest row's destPath (manifest.library). The RunPod network volume is a cache of it (manifest.cache, 400 GB, isis-model-cache): workers read from the cache, the cache is filled from the library. Pinned families (manifest.families[].pinned: the Chroma stack with its LoRAs, both Wan 2.2 expert pairs, the Wan shared encoders, SAM 3, BiRefNet, FlashVSR, GIMM-VFI, Real-ESRGAN — 212.26 GB) are never evicted; every other family stays until space is needed and then the family used longest ago leaves first. Shared files (umt5, the Wan VAE, clip_vision_h) leave only when no remaining family needs them, and a family a queued or running job leases is never evicted.

Adding a model. (1) A manifest row with families (lint node scripts/isis/runpod-volume-manifest-lint.mjs --strict must stay green, including the cache invariants it prints); (2) if a workflow uses it, add the family to that workflow's requires_families (model-families.spec.ts fails otherwise); (3) put the bytes in the library — library-pull streams from Hugging Face or Civitai with no local copy:

bash
node scripts/isis/runpod-provision.mjs pod create-loader --manifest <manifest-with-the-new-rows> \
  --mode library-pull --dc EU-RO-1 --parallel 8 --dry-run      # keys show as <n chars>
node scripts/isis/runpod-provision.mjs pod create-loader ... --yes

(4) the cache picks it up the first time a job needs its family. Nothing is copied to the cache by hand.

Loader modes (LOADER_MODE, scripts/isis/runpod-volume-loader.py): library-seed (volume → library), library-pull (source → library), library-verify (download and hash every row, write verifiedAt), library-probe (L.00.01 transfer measurement), cache-sync (a plan of evictions and pulls, applied to the mounted cache). Every mode writes /runpod-volume/.oshun/<mode>-result.json (read it back with runpod-volume-s3-put.mjs against the volume's S3 API) and prints a RESULT line; delete the pod afterwards — it sleeps an hour for that purpose and bills $0.06/h meanwhile. The two Hetzner keys are read from the env names in manifest.library.auth and injected as pod env; --dry-run redacts them. Single PUTs up to 512 MiB carry the manifest sha256 as the signed payload hash (the server refuses wrong bytes); above that, multipart with a per-part hash and a whole-file hash computed while reading — a mismatch aborts the upload. Hetzner's multipart works (measured 2026-09-13); RunPod's volume S3 multipart does not, which is why nothing uploads to the volume over S3 above 4 GiB.

Warming a session. Before a session that will use unpinned families (LTX-2.3, Animate, VACE, the controlnet_aux preprocessors, RMBG-2.0), warm them: POST /api/v1/runpod/cache/sync with {"workflows": ["ltx23-av"], "dryRun": true} shows the pulls, the evictions and the estimated wait; the same body without dryRun starts the sync (one per volume at a time; it bills a cache_sync line at the CPU pod rate, and the kill switch and budget guardrail apply). The dashboard's RunPod page has the same "Warm for session" picker.

409 models_not_cached. A job whose families are not cached is refused at submit, before any billing, with the families, their bytes, the estimated wait and the exact sync call to make. With ISIS_RUNPOD_CACHE_AUTOSYNC=true the job is held as awaiting_cache and the sync starts for it instead. When no cache volume is configured (ISIS_RUNPOD_CACHE_VOLUME_ID and manifest.cache.volumeId both unset) the gating is off and jobs run against the old full volume.

Pinning or unpinning a family. Flip pinned in manifest.families, run the lint (the invariants must still hold at the cache size), commit. Pinning does not pull anything by itself; the next sync that plans for it keeps it.

Licences (A.04.04). Every weight's licence, territory, acceptable-use clause, Civitai flags, content policy and distribution is in model-licences.md, generated from the manifest by node scripts/isis/model-licence-register.mjs (its test fails when the file drifts from the manifest). Adding a model edits the manifest row's licence string and regenerates the register in the same commit; the exclusions table names what must never be added.

Outputs retention and the storage alarm (L.07.02). Worker outputs under isis/outputs/ share the bucket, and the account's included allowance, with the library. The API's Hetzner storage guard sweeps outputs older than ISIS_HETZNER_OUTPUTS_RETENTION_DAYS (7) every ISIS_HETZNER_STORAGE_GUARD_INTERVAL_MS (6 h), then sums every bucket the credentials list and raises storage.hetzner.usage_high at ISIS_HETZNER_USAGE_ALARM_GB (900). The library prefix is refused by the S3 client itself, whatever the env says. Every over-the-line check logs ops.hetzner_storage.usage_high at error level; the webhook goes out once per UTC day with event id storage.hetzner.usage_high:<date>. The same two jobs from a shell:

text
node scripts/isis/runpod-volume-s3-put.mjs hetzner-usage            # exit 2 when over the line
node scripts/isis/runpod-volume-s3-put.mjs hetzner-sweep --days 7   # add --yes to delete

Verifying the library. library-verify over the manifest downloads and hashes every row. The old 1024 GB volume ax0oj250j3 was deleted on 2026-09-13 at 18:13Z on the user's explicit go (L.06.05), after both endpoints were repointed to 5bst2pgw7y and its loader result files were copied to docs/agents/evidence/isis-chroma-runpod/l-results/. The library is now the only full copy of the weights; a lost cache volume is rebuilt by syncing families from it.

6. Submitting from the API instead (mask-only flows, scripts)#

bash
curl -s -X POST http://127.0.0.1:4010/api/v1/jobs -H 'Content-Type: application/json' -d '{
  "type": "text-to-image",
  "prompt": "a red bicycle leaning on a white wall, morning light",
  "parameters": { "workflowId": "chroma-txt2img", "checkpoint": "chroma1-hd", "aspect": "square_1024", "seed": 42 }
}'
curl -s http://127.0.0.1:4010/api/v1/jobs/<jobId>

A refusal with RUNPOD_DAILY_BUDGET_EXCEEDED (429) or RUNPOD_COST_LEDGER_UNAVAILABLE (503) is the cost gate working, not a bug.

To run one catalog workflow straight at the endpoint, bypassing the API (this is what the C.11 matrix used), it is npx tsx, not node — the script imports catalog-renderer.ts at runtime and that module's extensionless imports are unresolvable to plain Node ESM, which fails as Cannot find module .../expression-evaluator:

bash
npx tsx scripts/isis/runpod-catalog-run.mjs matting-remove-background \
  --image source_image=docs/agents/evidence/isis-chroma-runpod/chroma-txt2img-0-9d49c5b4.png \
  --set model=rmbg-2.0 --out /tmp/run-out

It needs RUNPOD_API_KEY and RUNPOD_COMFYUI_ENDPOINT_ID only — no dev stack. It gives up after 15 minutes in IN_QUEUE, which on this endpoint is now a real possibility: see §2e.

6c. Getting a failed job's worker log (it is a stream, not a document)#

Three failures on 2026-09-12 — wan22-animate killing ComfyUI, flashvsr-upscale returning a bare Job processing failed, and ltx23-av before its cause was found in the graph — were all opaque for the same reason, and on 2026-09-13 that reason turned out to be a bug in the capture code, not the platform.

GET https://api.runpod.io/v2/serverless/<endpoint>/workers/<workerId>/logs answers content-type: text/event-stream: id: plus data: frames, one JSON object each ({"source":"system"|"container","line":…,"ts":…}), replayed from the worker's first line at roughly eight frames a second and then held open for live lines. It never closes. Both harnesses did await response.text() on it, which can never resolve, and their 20 s timeout then wrote "log fetch timed out" into every failure row. Measured live: ?tail=N is honoured (the last N lines, then live); Last-Event-ID and ?since= are not; once the worker is recycled the URL answers 404 "worker not found" in about three seconds — that part of the old explanation was right, and it still means a log is a race you can lose.

The reader is scripts/isis/runpod-worker-log.mjs: it asks for the tail, collects frames until the stream goes quiet for 2.5 s or a 20 s cap lands, and writes plain <ts> <source> <line> lines. A capped read is marked truncated rather than passed off as complete. Both runpod-catalog-run.mjs and motion-live-proof.mjs call it the moment a job comes back FAILED.

bash
node scripts/isis/runpod-worker-log.mjs --endpoint "$RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID" --worker <workerId> --tail 400
node scripts/isis/runpod-worker-log.mjs --endpoint "$RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID" --worker <workerId> --out docs/agents/evidence/isis-chroma-runpod/<run>/worker-<workerId>.log

The worker id is on the job's status response (workerId) and on GET /v2/serverless/<endpoint>/workers. status.output.errors[] remains the first place to look — it carried ltx23-av's Node 8 (LTXVImgToVideoInplace): required_input_missing 'image' with no log at all — but a Job processing failed with nothing in errors[] is exactly the case the log exists for.

6b. Reading a RunPod build log (when a git-integration build fails)#

The builds API tells you a build failed and almost nothing else:

bash
curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \
  "https://api.runpod.io/v2/serverless/$RUNPOD_COMFYUI_ENDPOINT_ID/builds" | jq '.[0]'
# -> "Command '['/usr/bin/docker', 'buildx', 'build', ...]' returned non-zero exit status 1."

There is no /logs route under it — every shape 404s. Use:

bash
node scripts/isis/runpod-build-log.mjs --latest          # the most recent build
node scripts/isis/runpod-build-log.mjs <buildId>          # a specific one
node scripts/isis/runpod-build-log.mjs --latest | grep -nE "ERROR|IMPORT FAILED|Traceback|error:"

It calls the same getBuildLogs GraphQL operation the console uses, with the ordinary API key, and reverses the result — the API returns newest-first, and a build log reads oldest-first. The first real error is near the top; the generic docker summary is at the end and is never the reason.

Do not skip this and change the Dockerfile on a hypothesis. Two builds were spent that way on 2026-09-12: the guess was import cupy on a GPU-less builder, the reality was a dangling symlink breaking ComfyUI-SAM3's os.makedirs, and the log said so plainly the whole time. It is the same lesson as C.12.12.

If you try the console instead: a RunPod console screenshot can come back completely blank while the page is fully rendered and interactive. Check document.body.innerText.length before concluding the page is broken. The log panel is also virtualised, so scraping the DOM only ever gets the ~15 lines currently on screen.

7. Teardown (every session)#

bash
for p in $(cat /tmp/claude-<session>.pids); do kill $p 2>/dev/null; done
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(4010|5180)'      # must be empty
rm -f /tmp/isis-chroma-env.sh
node scripts/isis/runpod-provision.mjs account            # pods 0, spend/h = volume only