Tracker: ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md (C.10). Evidence:
docs/agents/isis-chroma-runpod-evidence.md. Design: ADR-0005.
The generation API runs on this machine; the GPU work runs on the RunPod serverless endpoint (max one worker) against the network volume. Nothing here needs a local GPU.
0. Machine rules (CLAUDE.md, binding)#
memory_pressure | tail -1before every heavy start; below 35 % free → tear down first.- One Vite/tsx server per session. Track PIDs in
/tmp/claude-<session>.pids. - Servers die with the session:
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(4010|5180)'must be empty at the end.
1. Infra#
docker compose -f docker/docker-compose.dev.yml up -d postgres redis
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(5432|6379)\b' # both must listen
Postgres is postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev. The jobs
table is created on first boot (CREATE TABLE IF NOT EXISTS jobs). Redis backs
the daily USD ledger. MinIO is not needed until C.07 lands (outputs travel as
data URLs inside the job record until then).
2. Env recipe (throwaway, never committed)#
Keys come from ~/Desktop/workspace/drawer/env-master.env; copy by name, verify
by length only, delete the file at session end.
S=/tmp/isis-chroma-env.sh
{
grep -E '^(RUNPOD_API_KEY|RUNPOD_S3_ACCESS_KEY|RUNPOD_S3_SECRET_KEY|CIVITAI_API_KEY|GHCR_PULL_TOKEN)=' \
~/Desktop/workspace/drawer/env-master.env | sed 's/^/export /'
echo "export JWT_SECRET=$(openssl rand -hex 32)" # throwaway, never in env-master; see §2b
cat <<'EOF'
export NODE_ENV=development
export PORT=4010
export BASE_URL=http://127.0.0.1:4010
export DATABASE_URL=postgresql://oshun:oshun_dev@127.0.0.1:5432/oshun_dev
export REDIS_URL=redis://127.0.0.1:6379
export ISIS_QUEUE_PRODUCER_ENABLED=false # no BullMQ consumer exists locally
export ISIS_INLINE_JOB_RUNNER=true # the API runs its own jobs (C.06.06)
export COMFYUI_PROVIDER=runpod
export ISIS_RUNPOD_WORKER_KIND=worker-comfyui # API renders catalog → bare graph
export RUNPOD_COMFYUI_ENDPOINT_ID=6jye7vngg4i685 # rebuilt 2026-09-12 (V.03.04 note); see §2g
export RUNPOD_REGION=EU-RO-1 # provenance ledger `region` (the volume's datacenter)
# export ISIS_PROVENANCE_ALLOW_UNWATERMARKED_IMAGES=true # C.12.13 narrowed this
# and OFF is now the right default. Images are watermarked for real; this flag
# only decides what happens to the occasional output the stage could not mark
# (off = that job's result is refused at completion, the GPU time having
# already been spent). It no longer gates submission.
export ISIS_COST_RATE_RUNPOD_USD_PER_SECOND=0.000439 # RTX 5090 flex ($1.58/h) — the priciest
# card the rebuilt endpoint accepts, NOT the 4090's 0.000306. A gpuIds list means "any of
# these" and the ledger cannot know which card ran, so it has to bill the dearest one:
# under-billing is the direction that stops the budget firing. The preflight fails on this.
export ISIS_RUNPOD_DAILY_BUDGET_USD=3
export ISIS_RUNPOD_BUDGET_GUARDRAIL_ENABLED=true
export ISIS_RUNPOD_BUDGET_SOFT_LIMIT_USD=10
export ISIS_RUNPOD_BUDGET_HARD_LIMIT_USD=25
export ISIS_RUNPOD_BUDGET_THROTTLE_UNITS_PER_WINDOW=3
export ISIS_RUNPOD_BUDGET_THROTTLE_WINDOW_SECONDS=600
export ISIS_RUNPOD_BUDGET_FAIL_CLOSED=true
export ISIS_COST_RATE_DEFAULT_USD_PER_SECOND=0.000306
# export ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS=3600000 # C.12.17; default 3600000 (1 hour)
# The UNBILLED queue + cold-boot allowance, held separately from the GPU
# execution cap so a slow image pull cannot get a rendering job cancelled and
# re-billed. **Raised from 10 min to 1 HOUR on 2026-09-12.** Ten minutes was
# set when we thought a cold start's delayTime was billed, so waiting looked
# expensive; V.11.01 measured that it is not (ledger $1.20, account $0.29).
# Waiting is free, and EU-RO-1 needed FOUR attempts to place one clip — twice
# a card arrived within a minute of a shorter deadline. Lower it (e.g. 45000)
# for the kill-switch drills C.11.12 / V.11.04, which measure exactly this
# wait; restore afterwards. A zero or malformed value is ignored, not obeyed.
export VITE_ISIS_GENERATION_API_URL=http://127.0.0.1:4010
export CORS_ORIGINS=http://127.0.0.1:5180 # the API rejects the dashboard's XHR without it
# export ISIS_RUNPOD_BUDGET_ALARM_PROJECT_ID=<project id> # C.04.06: project whose webhook
# subscriptions receive runpod.budget.warning / .exhausted. Unset = alarms log only.
#
# --- Video extension (V.00.02). Commented out until V.03.04 creates the second
# --- endpoint: an unset id is a loud failure, an empty one is a silent
# --- submission to nowhere. Uncomment the block as a whole, id first.
export RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID=2lezocymx2rgot # V.03.04, created 2026-09-12; `motion` workflows route here
export ISIS_COST_RATE_RUNPOD_VIDEO_USD_PER_SECOND=0.001647 # H200 141 GB flex ($5.93/h) — the
# dearest card in the EU endpoint's list (user decision 2026-09-13; 0.002400
# was the deleted US endpoint's B200). A gpuIds list means "any of these" and
# the ledger cannot know which card ran, so it bills the dearest; the card that
# usually runs (RTX PRO 6000 Blackwell) settled at $0.00105/s on 2026-09-13.
# --- Model cache (option B, L.04). Leave ISIS_RUNPOD_CACHE_VOLUME_ID unset until L.06.01 creates the
# --- 400 GB cache volume: unset, cache gating is off and jobs run against the old full volume.
# export ISIS_RUNPOD_CACHE_VOLUME_ID=<isis-model-cache volume id>
export ISIS_RUNPOD_CACHE_SYNC_MB_PER_SECOND=40 # MiB/s; L.06.02 median with three transfers sharing Hetzner (about 75 alone; the L.00.01 probe read 20.8)
# export ISIS_RUNPOD_CACHE_AUTOSYNC=true # hold jobs as awaiting_cache instead of 409
# --- Hetzner storage guard (L.07.02): outputs retention + the 900 GB account alarm. Off unless enabled,
# --- and the sweep is a dry run until the second line says false. Needs the HETZNER_APP_DEV_* names.
# export ISIS_HETZNER_STORAGE_GUARD_ENABLED=true
# export ISIS_HETZNER_OUTPUTS_RETENTION_DRY_RUN=false # only after a dry-run log line looks right
# export ISIS_HETZNER_STORAGE_ALARM_PROJECT_ID=<ops project> # falls back to ISIS_RUNPOD_BUDGET_ALARM_PROJECT_ID
export ISIS_WORKER_OUTPUT_FETCH_HOSTS=hel1.your-objectstorage.com # C.12.06: hosts the API may
# fetch presigned worker outputs from (the bucket the worker's uploader writes
# to). Unset = only the RunPod volume's own S3 URL is accepted.
# export ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6 # second ledger key, separate from the image one
# export ISIS_RUNPOD_VIDEO_INPUT_INLINE_MAX_BYTES= # V.05.01 measures it; above this, inputs go to the volume over S3
# export RUNPOD_VOLUME_ID=5bst2pgw7y # the model cache volume, the only one since 2026-09-13
# export RUNPOD_S3_ENDPOINT=https://s3api-eu-ro-1.runpod.io # SigV4 host for that bucket
EOF
} > "$S"; chmod 600 "$S"; source "$S"
echo "RUNPOD_API_KEY len=${#RUNPOD_API_KEY} S3 len=${#RUNPOD_S3_ACCESS_KEY}/${#RUNPOD_S3_SECRET_KEY} civitai len=${#CIVITAI_API_KEY} ghcr len=${#GHCR_PULL_TOKEN} jwt len=${#JWT_SECRET} endpoint len=${#RUNPOD_COMFYUI_ENDPOINT_ID} video-endpoint len=${#RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID}"
2c. What the first live run (2026-09-11) made mandatory#
-
Consent id on every job. The provenance ledger (active whenever
DATABASE_URLis set) refuses to complete a generation withoutconsentId; the API now refuses such jobs at submit, before billing. The Studio's Chroma presets attachconsentId: consent_operator_isis_dev_studio(operator's own consent for a typed prompt); curl submissions must passparameters.consentIdthemselves. -
Images are watermarked (C.12.13, was: unwatermarked by policy). The worker-comfyui path now embeds a provenance payload —
isis:v1;job=…;consent=…;wf=…;model=…;produced=…— in the blue channel's low bit of every inline PNG output, before the bytes are persisted, and recordssha256(payload)as the bundle'swatermarkHash. The payload is framed with a CRC and repeated (512 times over a 1024² image), so damage to part of the picture still yields it back. Measured on the C.11 fixture: +1.9% file size versus ComfyUI's own encoder, +0.5 s, every sample moved by at most 1.What it does not do, on purpose: spatial LSB survives a lossless re-save and the storage path, but not JPEG/WebP re-compression, resizing, cropping or rotation. It is a provenance marker for the artifact Isis stores and serves, not a forensic watermark against a determined adversary; a transform-domain (DWT-DCT) version is separate, larger work.
ISIS_PROVENANCE_ALLOW_UNWATERMARKED_IMAGESsurvives with a narrower job and should be off: it decides only what happens to an output the stage could not mark. Off refuses that job's result at completion; on keeps it withwatermark: { applied: false, reason }recorded. A job with no still image to mark (a clip, a mask-only flow) is not covered by it either way. -
Runtime, settled 2026-09-11 21:39Z. The endpoint is
6l1f3tku7z2zoy(isis-comfyui-chroma), built by RunPod's git integration from branchrunpod-comfyui-slim. Its active released677f914(commit105ad3cf) works — it creates containers, boots ComfyUI and renders. The interim stock-image endpointac988l8is5axkfthat produced the first image was deleted and is no longer needed; templatezxojew9tyosurvives as a fallback only. -
If the endpoint stops placing workers, re-save it in the console. After a GraphQL
saveEndpointcall this endpoint went dead: zero workers in every state with jobs queued, and a RESTPATCH {"workersMax":1}did not revive it. Manage → Edit endpoint → Save Endpoint, re-applying the same configuration, fixed it immediately. Never call GraphQLsaveEndpointon a git endpoint; if one is already wedged, the console form is the cure, not a rebuild. -
"Low Supply" is not a blocker. The edit dialog labels EU-RO-1's selected pools Low Supply / Unavailable, yet a worker was placed within ~40 s. Do not conclude a capacity problem from that label alone, and do not propose migrating datacenters over it — a network volume cannot move, so migrating costs a second 500 GB volume plus re-seeding ~70 GB.
-
Cheapest possible liveness probe (costs nothing if no worker boots, ~$0.03 if one does): submit a model-free graph and purge afterwards.
bashcurl -s -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \ -d '{"input":{"workflow":{ "3":{"class_type":"EmptyImage","inputs":{"width":64,"height":64,"batch_size":1,"color":8421504}}, "4":{"class_type":"SaveImage","inputs":{"images":["3",0],"filename_prefix":"smoke"}}}}}' \ "https://api.runpod.ai/v2/$RUNPOD_COMFYUI_ENDPOINT_ID/run" # then: POST https://api.runpod.ai/v2/<ep>/purge-queue -
GPU pools, not GPU names. EU-RO-1 (the volume's datacenter) had no free 4090 and does not offer A5000/3090; both endpoints use pools
ADA_24,AMPERE_24,BLACKWELL_32set through GraphQLsaveEndpoint(the REST PATCH silently ignoresgpuTypeIds). Renders land on RTX PRO 4500 Blackwell. -
Measured cost. Cold run: 53 s queue/boot + 150 s GPU = $0.083 at the 4090 rate; the model load from the volume (T5 ~40 s, Chroma1-HD ~65 s) is most of the GPU time. Ledger rate stays 0.000306 $/s until RunPod's invoice for the Blackwell pool is compared (C.11.13).
Restarting the API mid-job is survivable now (C.12.16)#
A job that was in flight when the API process died used to be lost outright: the
inline runner claims only pending/queued, so the row sat at running
forever while RunPod finished the render and billed for it (job 7d4280f9,
148.98 s, gone). Two things changed:
- The RunPod job id is written to
job.metadatathe instant RunPod accepts the submission, before the first status poll —providerJobId,providerEndpointId,providerDispatchedAt. - The runner sweeps once at startup, before it claims anything. A
runningjob whose owning process is provably gone (inline-<pid>and the pid is dead, or the pinnedISIS_INLINE_JOB_RUNNER_IDis this process's own) is put back in the queue with its dispatch record intact. The next claim re-attaches to the existing RunPod job rather than submitting a new one, so nothing is rendered or billed twice, and the cost ledger is idempotent per job id anyway.
What it does NOT do, on purpose: a job whose worker id cannot be proved dead is
left exactly where it is, and logged as left with the reason. Two processes
running the same job would double-bill it, which is worse than a stuck row. A
job that has no providerJobId (dispatched before this change, or the write
failed) is failed with ORPHANED_WITHOUT_PROVIDER_JOB_ID rather than left lying
about being running — nothing can say which render was its.
Watch the startup log for Re-queued an orphaned job for re-attachment and
Re-attaching to an already-dispatched RunPod job instead of submitting. Note
the CLAUDE.md rule still stands for its own reasons: do not edit app source
while a run is in flight, because tsx watch recompiles under it.
2e. Cold boot on this endpoint got much slower when the video packs landed#
Measured 2026-09-11, before the video packs: a fresh EU-RO-1 host pulled the
14.7 GB worker image in ~4 minutes (job d1e87d06).
Measured 2026-09-12, after V.01.01 added six video node packs and their
dependencies (cupy, triton, pycocotools, timm, the controlnet_aux annotators): a
fresh host was still initializing after 15 minutes, and
runpod-catalog-run.mjs gave up in IN_QUEUE without the job ever starting.
Nothing was billed — delayTime is not GPU time — but nothing ran either.
What this means in practice:
- A cold start here is minutes, plural, and possibly many. Plan a session
around one boot: submit the first job, let the worker come up, then run
everything else back to back while it is warm (
idleTimeoutis 10 s, so "warm" means within seconds of the previous completion, not minutes). initializingis not the same as wedged. The endpoint fault C.12.12 recorded shows zero workers in every state with jobs queued. One workerinitializingmeans placement worked and the host is pulling the image. Do not re-save the endpoint, do not rebuild, and do not resubmit — the queued jobs run when the boot finishes.- A queued job survives the client giving up. The runner's 15-minute cap
abandons the wait, not the job. Read it back later with
GET /v2/<endpoint>/status/<jobId>; do not submit a second one, which just queues behind the first.
2d. Video extension (V.10.01)#
Nothing here is needed for the image stack. Turn it on as a block — a
half-configured video endpoint is the dangerous state, not an unconfigured one,
which is why dev-stack-preflight.mjs skips every video check while
RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID is empty and binds all of them the moment it
is set.
-
RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID— the second endpoint (V.03.04 prints it on creation;scripts/isis/runpod-provision.mjs endpoint listrecovers it later). Everymotion-category workflow routes here. There is no fallback to the image endpoint — amotionsubmit with this unset fails loud (V.06.03). Both endpoints run the same image off the same volume; the difference that matters is the card, and falling back would send a 28 GB expert to a 24 GB pool and bill the OOM (comfyui.service.ts:967). -
ISIS_COST_RATE_RUNPOD_VIDEO_USD_PER_SECOND=0.001647— H200 141 GB flex ($5.93/h), not the H100's 0.001331. The endpoint'sgpuIdslist includes H200, and a list means "any of these": the ledger cannot know which card ran, so it has to bill the dearest or it under-counts by 24% every time an H200 is assigned. (It carried 0.001331 until 2026-09-12 because the preflight's price table had no H200 row and reported it UNPRICED rather than guessing — correct behaviour that still left the rate wrong; the table now prices all seven enabled cards from §2f's measured tiers.) Copying the IMAGE rate here is the other mistake to fear: an H200 second costs ~3.8x a 5090 second, so the ledger would under-count by that factor and the budget would never fire. The preflight asserts this against thecomfyui-videoentry'sgpuIdsininfra/runpod/endpoints/desired-state.jsonand fails if the configured rate is below the priciest GPU that endpoint may be assigned (0.000486 if V.11.06 moves Wan to L40S). -
ISIS_RUNPOD_VIDEO_DAILY_BUDGET_USD=6— a second ledger with its own Redis key (:video), separate from the image one. A single video job can cost more than the entire image day budget: a 5-second 720p Wan 2.2 clip is a few minutes of H100 time, and the image budget is $3/day total. The preflight caps this at 50 for the same reason. -
Volume S3 (
RUNPOD_S3_ACCESS_KEY,RUNPOD_S3_SECRET_KEY,RUNPOD_VOLUME_ID=5bst2pgw7y(the model cache;ax0oj250j3was deleted 2026-09-13),RUNPOD_S3_ENDPOINT=https://s3api-eu-ro-1.runpod.io, andBUCKET_ENDPOINT_URLfor the worker side). Inputs and outputs above the inline ceiling ride the volume rather than the request body. The preflight checks the two credentials by LENGTH only and never prints them, and refuses aBUCKET_ENDPOINT_URLwhose host disagrees withRUNPOD_S3_ENDPOINTor whose path does not name the volume — pointing the worker's bucket anywhere else writes outputs where nothing reads them. -
Retention sweep (
ISIS_VOLUME_RETENTION_SWEEP_ENABLED,ISIS_VOLUME_RETENTION_DRY_RUN,ISIS_VOLUME_RETENTION_DAYS,ISIS_VOLUME_RETENTION_SWEEP_INTERVAL_MS). The API can deleteinputs/<jobId>/andoutputs/<jobId>/objects past an age, so the volume does not get grown to store last month's test clips. It is off unless enabled and a dry run even then — setISIS_VOLUME_RETENTION_DRY_RUN=falseto actually delete, and read the dry-run log line first; it is the same line the real run prints.models/is not sweepable at any age and the refusal is in the client, not the config. By hand:node scripts/isis/runpod-volume-s3-put.mjs sweep --days 7(add--yesto delete).
ffprobe is a convenience here, not a dependency. The API reads a clip's
real duration, fps, frame count and dimensions out of the MP4 boxes itself
(apps/isis/generation-api/src/services/mp4-probe.ts), precisely because
ffprobe is silently absent on a slim image and a stack that depends on it
fails in a way nobody notices. Install it anyway for cross-checking a suspicious
clip by hand:
ffprobe -v error -select_streams v:0 \
-show_entries stream=width,height,nb_frames,avg_frame_rate \
-show_entries format=duration -of default=nw=1 out.mp4
# no output at all -> not installed; `brew install ffmpeg` (macOS),
# `apt-get install -y ffmpeg` (Linux dev server). Nothing in the API needs it.
Two honest caveats when comparing its numbers to the API's: mp4-probe reports
the AVERAGE fps over the track (frames / track duration), so a clip truncated by
-shortest muxing reads lower than its nominal rate — measured at 12.8 vs 16 on
such a file, while every untruncated clip (5/9/17/33/81 frames at 16 fps) agrees
exactly. And WebM is a different container (EBML): the parser returns null for
it rather than guessing, so an alpha .webm keeps whatever the graph declared.
2f. Stopping the video endpoint, and what happens when there is no GPU (V.03.05)#
Endpoint 2lezocymx2rgot (isis-comfyui-video), EU-RO-1, workersMin 0 /
workersMax 1, 900 s execution cap.
Kill switch — workersMax to 0. Either the dashboard's per-endpoint button
(runpod-kill-switch.service.ts, target video) or:
node scripts/isis/runpod-provision.mjs endpoint update 2lezocymx2rgot --workers-max 0 --yes
node scripts/isis/runpod-endpoint-drift-check.mjs --key comfyui-video # expect DRIFT on workersMax
A bare runpod-endpoint-drift-check.mjs with no --key now checks every
endpoint in the environment and says how many it checked; --key narrows it to
one.
A paused endpoint queues NOTHING — it refuses the submit. Measured against
the live video endpoint at workersMax: 0 (V.11.04, 2026-09-12), /run
answers:
{
"status": 409,
"title": "Conflict",
"code": "ENDPOINT_PAUSED",
"detail": "Endpoint is paused (max_workers=0). Set max_workers > 0 to accept work."
}
/health stayed at inQueue: 0 with no worker in any state. So the switch is
stronger than "nothing new starts": there is no backlog waiting to stampede when
the ceiling goes back up, and nothing to cancel. The API surfaces it as
RUNCOMFY_ENDPOINT_PAUSED with the message "The RunPod endpoint is stopped
(workersMax is 0) and is accepting no work. Raise workersMax in the RunPod
console to resume.", and does not retry it — a paused endpoint stays paused
until a human raises it, so every retry would be another 409. (It arrived as
"Unexpected RunPod provider error." until this was fixed.)
Raising it again is deliberate and manual. Two more things measured 2026-09-12:
/runkeeps answering409 ENDPOINT_PAUSEDfor ~30 s after you raiseworkersMaxagain. REST reports the new value immediately. That lag is propagation, not the wedged-endpoint fault — wait half a minute and retry before diagnosing anything.- A job that DID reach the queue before the pause outlives the client that
submitted it.
runpod-catalog-run.mjsgives up after 15 minutes; the job does not. Read it back withGET /v2/<endpoint>/status/<jobId>rather than submitting a second one, which only queues behind the first.
No GPU available: the job waits, and then the API gives up. It does NOT fall back to a smaller card. That refusal is deliberate — a 28.6 GB fp16 Wan expert that OOMs has already been billed for the load, so a "helpful" fallback to a 48 GB card converts a wait into a charge for nothing. The endpoint is pinned to cards that all clear the 70.53 GB peak (three RTX PRO 6000 Blackwell variants, three H100 variants, H200) and nothing smaller is enabled.
While the job sits IN_QUEUE nothing is billed — RunPod's delayTime is not
GPU time — so the cost of waiting is wall clock only. What ends the wait is the
queue/boot allowance from C.12.17 (ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS, default
3 600 000 — one hour), after which the API cancels the job as retryable
because nothing was spent.
Every queue-side deadline is one hour as of 2026-09-12, and they are a different thing from the execution caps. Queue wait is unbilled, so a short deadline buys nothing and throws away work that was about to run:
| Deadline | Value | Where |
|---|---|---|
| API queue/boot allowance, per attempt | 1 h | ISIS_RUNPOD_QUEUE_BOOT_TIMEOUT_MS |
| Matrix runner give-up (cold start only) | 1 h | motion-live-proof.mjs |
| Single catalog run | 1 h | runpod-catalog-run.mjs |
| Orphan collector | 1 h | motion-proof-collect.mjs --deadline-min |
| Execution cap, image | 5 min | ENDPOINT_EXECUTION_TIMEOUT_MS.image |
| Execution cap, video | 15 min | ENDPOINT_EXECUTION_TIMEOUT_MS.video |
The last two bound BILLED GPU time and are the cost guard. Do not raise them to match the queue ones.
Supply in EU-RO-1 is the real constraint, and it is not what the console shows you first. Measured 2026-09-12, the same three tiers before and after attaching the network volume:
| Tier | Before volume (global) | After volume (EU-RO-1) |
|---|---|---|
| 96 GB, $3.49/h | High Supply | Low Supply |
| 80 GB, $4.79/h | High Supply | Unavailable |
| 141 GB, $5.93/h | Medium Supply | Unavailable |
So the three-deep fallback is, in practice, a chain of one. All three stay enabled because "Unavailable" is a point-in-time reading rather than a property, but plan on 96 GB or a wait. Never size a RunPod endpoint from the labels shown before the volume is attached — they describe datacenters you will not be deployed in.
Re-measured 18:20Z the same day, with every tier the console offers, because
V.11.01 could not get a card at all (three attempts, throttled, $0.00
billed, and the console saying "There's no free GPU capacity for your current
selection"):
| Tier | Enabled | Supply | $/hr |
|---|---|---|---|
| 96 GB Pro | 1st | Low Supply | 3.49 |
| 80 GB Pro (H100) | 2nd | Unavailable | 4.79 |
| 141 GB (H200) | 3rd | Unavailable | 5.93 |
| 80 GB (A100) | 4th | Unavailable | 2.72 |
| 16 GB | no | Medium Supply | 0.58 |
| 24 GB Pro | no | Low Supply | 1.10 |
| 24 / 32 / 48 GB | no | Unavailable | ≤ 1.75 |
| 180 GB Pro | no | Unavailable | 8.64 |
| 288 GB Pro | no | Unavailable | 10.65 |
Every tier that can hold a 70.53 GB Wan expert is Unavailable except the 96 GB
Pro one, which is Low Supply. Adding the two A100 80 GB variants (done the
same day, endpoint version 2) did not help: that tier is dry too. There is no
card selection that fixes this — the only ≥80 GB tier with any supply is the one
that was already first in priority. If a clip has to run, the region is the
only lever left; see §2g. Note also that a worker can reach ready and be
taken back before it picks up a job (observed 18:14:40Z → 18:15:00Z), so a
ready reading is not yet capacity.
Two practical consequences when supply IS found:
- Nothing is billed while a job waits or a worker is
throttled— the console's billing panel read$0.00across all three attempts. Waiting costs wall clock only, and the C.12.17 queue-phase timeout retries it for free. - The 10 s
idleTimeoutis dangerous in a low-supply region: lose the worker between two clips of a warm session and the next one rejoins the same queue with no guarantee of a card. Raise it for the duration of a multi-clip session (V.11.02) rather than paying for a second placement gamble.
2g-migration. Moving to US-CA-2 (in progress 2026-09-12)#
Closed 2026-09-12 23:58Z — the stack stays in EU-RO-1. User decision after the §2g-full-survey below: no region has proven supply of both card tiers, US-CO-1 is H200-only, and US-CA-2 never placed a worker. Everything outside EU-RO-1 was deleted in one pass (endpoints
eron7rmrmcg0rq,ajcpag53ed052j,ay4gcv868ein1u; their templates plus the seven probe templates; volumessq5230axbcUS-CA-2 andfj3f77pca3US-CO-1). What remains is exactly the pre-migration state: volumeax0oj250j3, endpoints2lezocymx2rgotand6jye7vngg4i685. The recipe below is kept because it is correct for any future move — read the survey and the region-hunt rules first.
Approved after the probe below. Nothing in EU-RO-1 is touched until a clip has rendered in US-CA-2 — the 500 GB volume there is the only copy of 358 GB that took hours to fetch, and a half-finished migration that deleted it would be unrecoverable.
# 1. Volume in the new region. A volume PINS the datacenter, which is also why
# the existing endpoints cannot follow it — new ones are required.
curl -s -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
-d '{"name":"isis-chroma-models-us","size":500,"dataCenterId":"US-CA-2"}' \
https://rest.runpod.io/v1/networkvolumes # -> sq5230axbc
# 2. Loader pod. CPU-only ($0.06/h); V.02.04 forbids a GPU pod for this.
node scripts/isis/runpod-provision.mjs pod create-loader \
--volume sq5230axbc --dc US-CA-2 --yes # -> mf35u5m5k0c8xi
# 3. Watch it. The .part suffix is the loader resuming with HTTP Range.
export RUNPOD_S3_ENDPOINT=https://s3api-us-ca-2.runpod.io
node scripts/isis/runpod-provision.mjs volume ls sq5230axbc --dc US-CA-2
# 4. The verdict is the loader's own, not a size check: it verifies every
# sha256 and exits 0 only if all 42 rows are "ok".
node scripts/isis/runpod-provision.mjs volume cat sq5230axbc .oshun/manifest-result.json --dc US-CA-2
# 5. Delete the pod the moment the pull is done — it bills while it lives.
node scripts/isis/runpod-provision.mjs pod delete mf35u5m5k0c8xi --yes
A CPU pod cannot run the worker image. POST /pods with computeType: CPU
rejects containerDiskInGb above 20, and both live templates for the video
image declare 30. A CPU pod created at 20 GB sat 13 minutes with runtime: null
(pull never finished) before being deleted. Anything that needs the image's own
filesystem — the build-time object_info.json in /opt/oshun/build-manifest/,
for one — has to run as a serverless GPU worker on the same volume.
Set workersStandby: 0 explicitly on the new endpoints. Creating a
serverless endpoint through the REST API defaults it to 3 (and workersMax to
3), and a standby worker starts billing with no job submitted — see the
probe note below. The two EU endpoints carry workersStandby: 1 and are not
billing for it only because EU-RO-1 cannot place a worker at all.
Order of the cutover, deliberately conservative: pull → verify all 42 sha256s → create both endpoints on the new volume → render one clip → only then delete the EU volume and its endpoints. The EU stack keeps serving throughout.
Everywhere the region is pinned. Found by grep rather than memory, because a missed one fails as "no GPU available" or a silently wrong provenance record rather than as an error:
| File / setting | Now | At cutover |
|---|---|---|
infra/runpod/volumes/isis-chroma-models.manifest.json → volume.dataCenterId |
EU-RO-1 |
US-CA-2 |
…same file → volume.name |
isis-chroma-models |
isis-chroma-models-us |
infra/runpod/endpoints/desired-state.json → both dataCenterId |
EU-RO-1 |
US-CA-2 |
…both networkVolumeId |
ax0oj250j3 |
sq5230axbc |
…both endpointId |
EU ids | the new ids |
…both workersStandby |
not declared (RunPod default) | 0, explicitly |
Env recipe RUNPOD_REGION |
EU-RO-1 |
US-CA-2 — it is the provenance bundle's region, so a stale value mislabels every render |
Env recipe RUNPOD_S3_ENDPOINT |
https://s3api-eu-ro-1.runpod.io |
https://s3api-us-ca-2.runpod.io |
Env recipe RUNPOD_VOLUME_ID |
ax0oj250j3 |
sq5230axbc |
| Env recipe both endpoint ids | EU ids | the new ids |
The manifest is shared between the two volumes while both exist, so its
volume block is changed at cutover and not before — editing it early would
make the lint describe a volume that is still serving.
RUNPOD_REGIONS in libs/isis/runpod-surface/src/endpoint-registry.ts is an
AWS-style list (us-east-1, …) unrelated to RunPod's datacenter ids and needs
no change; RUNPOD_REGION is a free string read only for the provenance
region.
The S3 listing returns intermittent HTTP 524. Two tries in three failed while the loader was writing. It is Cloudflare timing out on the listing, NOT an empty volume — retry before believing a zero, and never report an error as 0 GB.
2g-region-hunt. US-CO-1 grants cards; US-CA-2 and EU-RO-1 do not (2026-09-12 23:2xZ)#
How to test a region, since the obvious routes do not work. The console's
supply labels need a volume to pin the datacenter and are far too slow to read a
dozen times. GraphQL introspection is disabled.
lowestPrice(input:{dataCenterId}) returns almost nothing per-DC and quotes
B200 at the same $5.98 everywhere — it is the POD market, not serverless. So
test placement, which is the ground truth anyway and is pure API:
- Create a 10 GB volume in the datacenter (a volume is what pins it).
- Create a throwaway endpoint on it with the cards you actually want, and a tiny public image so nothing waits on a pull.
- Read
/v2/<id>/healthand countready + idle + running. workersMax: 0, then delete endpoint → then volume (a volume attached to a live endpoint returns 500).
initializing is NOT evidence of a card. It is equally what a worker shows
while WAITING for one — the Workers tab spells it out: "Waiting for GPU to
become available…". Only ready/idle/running proves a card was granted.
Counting initializing as success will tell you a starved region is healthy.
Measured, same eight large cards offered to all four:
| Datacenter | Card granted? | Evidence |
|---|---|---|
| US-CO-1 | yes, repeatedly | placed=2 at 23:21, 23:22 and 23:23 |
| US-GA-2 | once, not repeatable | ready/idle on the first read only |
| US-CA-2 | no | 1 h+ queued; the V.11.02 verification run gave up having never started, $0 |
| EU-RO-1 | no | initializing all session, never placed |
The account caps at 5 max-workers across all endpoints, so probes have to be run in batches and torn down between them.
One placement is not evidence. US-CA-2 was migrated to on the strength of a single worker appearing within a minute at 20:02Z, and it has not granted a card since — the 358.30 GB now sits in a region that cannot serve it. US-CO-1 is recorded here only because it held across three consecutive readings. Apply that bar before moving anything again.
2g-full-survey. Every storage-capable datacenter, one form read (2026-09-12 23:50–23:53Z)#
The question that should have been asked before any volume was created: which
regions have the card tiers BOTH endpoints need? Read from the console's
deploy form with exactly one datacenter selected and no volume, for all
seventeen datacenters GraphQL marks storageSupport: true — a volume pins the
region, so nowhere else can host the stack. Raw labels and prices:
docs/agents/evidence/isis-chroma-runpod/region-supply-survey-2026-09-12T2350Z.json.
Image needs 24 / 24 PRO / 32 PRO ($0.69 / $1.10 / $1.58 per hour). Video needs
80 PRO (H100, $4.79), 96 PRO (RTX PRO 6000, $3.49), 141 (H200, $5.93) or 180 PRO
(B200, $8.64). — is Unavailable.
| Region | 24 GB | 24 PRO | 32 PRO | 80 PRO | 96 PRO | 141 GB | 180 PRO |
|---|---|---|---|---|---|---|---|
| EU-RO-1 | Low | Medium | Low | — | Low | — | — |
| US-CA-2 | Low | Low | — | Low | Low | Low | Low |
| US-CO-1 | — | — | — | — | — | Low | — |
| US-GA-2 | Low | — | — | Low | — | Low | — |
| US-NC-2 | — | — | — | — | Low | — | Low |
| EUR-IS-1 | Low | — | — | — | Low | — | — |
| CA-MTL-3 | — | Low | — | — | Low | Low | — |
| AP-JP-1 | — | — | — | Low | — | Low | — |
| EU-FR-1 | — | — | — | Low | — | Low | — |
| EUR-IS-3 | — | — | — | Low | — | — | — |
| EUR-NO-1 | — | Low | Low | — | — | — | — |
| EUR-NO-2 | — | — | — | Low | — | — | — |
| US-IL-1 | Low | High | — | — | — | — | — |
| US-MO-2 | Low | — | — | — | — | — | — |
| US-TX-3 | — | Low | — | — | — | — | — |
| CA-MTL-4 | — | — | — | — | — | — | — |
| EU-NL-1 | — | — | — | — | — | — | — |
What it settles.
- US-CO-1 is an H200-only datacenter. Three independent readings agree: this
form, the earlier console table in §2f, and GraphQL
dataCenters.gpuAvailability(which lists exactly one card there). The workers the region hunt saw placed can only have been H200s. So US-CO-1 is the most expensive place to run video short of B200 ($5.93/h — 1.7× the 96 GB PRO rate, 2.2× the A100 the EU clip actually billed at), and it has nothing at all for the image endpoint: the chroma stack could not follow the volume there. The region hunt measured "does a worker get placed" and never asked "on what card" — that is the gap this table closes. - No region has proven supply of both tiers. Five carry labels for both (EU-RO-1, US-CA-2, US-GA-2, EUR-IS-1, CA-MTL-3); only EU-RO-1 has ever rendered anything for us, and US-CA-2 — the widest label coverage in the table — never placed a worker in three hours. A Low label does not predict placement in either direction: US-CA-2 had four Low tiers and placed nothing; US-CO-1 had one and placed repeatedly. Placement tests (§2g-region-hunt) remain the only ground truth, and they must now also record the card granted.
- EU-RO-1 is the best image region on the board — Medium on 24 PRO, the only Low-or-better 32 PRO besides EUR-NO-1, and every image job has rendered there. The image endpoint has no reason to move.
- Candidates for video that are untested: US-NC-2, CA-MTL-3 and EUR-IS-1 list 96 GB PRO. Each needs the placement test with the video endpoint's exact card list before a byte is copied — the US-CA-2 migration is what happens when that step is skipped.
Three volumes are billing right now (EU-RO-1 ax0oj250j3, US-CA-2
sq5230axbc, US-CO-1 fj3f77pca3; $35/month each at 500 GB). US-CA-2 has never
served a job and US-CO-1 is empty; at most one of the two should survive the
decision.
US-CO-1 has no pod market and no S3 API. POST /pods rejects it (the REST
dataCenterIds enum predates the region; without the field, every CPU flavour
answers "no instances available"), and s3api-us-co-1.runpod.io does not
resolve. The only way to write to a volume there is a serverless worker in the
region — a CPU-serverless loader endpoint (ay4gcv868ein1u, template
e2gcmznv3p, python:3.12-slim running the C.02.06 loader behind a tiny
runpod handler that returns manifest-result.json as job output) was created
for that and parked at workersMax: 0 pending this decision.
2h. The object-info snapshot comes out of the build log#
infra/runpod/endpoints/node-classes/worker-comfyui-5.10.0-video.json is the
worker image's own /object_info, reduced to what the graph validator reads,
and it is refreshed like this — no GPU, no pod, no S3:
git push origin HEAD:runpod-comfyui-slim # RunPod rebuilds; smoke emits OBJINFO lines last
node scripts/isis/runpod-build-log.mjs --latest --endpoint 2lezocymx2rgot # wait for COMPLETED
node scripts/isis/runpod-build-snapshot.mjs --endpoint 2lezocymx2rgot # verifies chunks + sha256, writes the file
npx vitest run --root libs/isis/workflows graph-validator # every golden against it
Why the log: getBuildLogs takes only buildId and startTime and returns the
newest 500 lines, so the smoke prints the snapshot as the very last thing (75
lines of 1600 chars for 1071 classes). A CPU pod cannot hold the image
(containerDiskInGb ≤ 20, the image needs 30), US-CO-1 has no S3 API, and a GPU
worker in EU-RO-1 waited over an hour without a card. The root .dockerignore
excludes infra; the manifest is re-included by name so the smoke can shadow
the volume — a new file under infra/ that the build needs must be re-included
the same way.
2g-supply. Both regions fluctuate; neither is reliably warm#
The honest comparison, each read off the endpoint's own GPU configuration list:
| Tier | EU-RO-1 @ 18:20Z | US-CA-2 @ 22:55Z |
|---|---|---|
| 96 GB Pro, $3.49/h | Low Supply | Unavailable |
| 80 GB Pro H100, $4.79/h | Unavailable | Low Supply |
| 141 GB H200, $5.93/h | Unavailable | Low Supply |
| 80 GB A100, $2.72/h | Unavailable | Unavailable |
| 180 GB B200, $8.64/h | Unavailable | Low Supply |
US-CA-2 has two of the enabled large tiers at Low Supply where EU-RO-1 had one, plus B200 — which is why the move is defensible. But at 22:55Z US-CA-2 was ALSO answering "There's no free GPU capacity for your current selection" with two jobs queued. Neither region is reliably warm, and a single placement is not evidence that one is. The probe that triggered this migration got a card in US-CA-2 in under a minute at 20:02Z; the same selection could not get one three hours later. Read the per-tier table before concluding anything about a region.
Initializing does not mean "pulling the image". It is also what a worker
shows while it waits for a card it has not been given. The Workers tab says
which — "Waiting for GPU to become available…" — and the Logs tab stays empty
("logs will appear here once they're ready"). Do not infer a pull from the
status string.
B200 is enabled on the US video endpoint (2026-09-12) because it was the
only large tier showing supply there and it is the one plausible shot at
wan22-animate, which died at the floor of the caps and may never have had a
card big enough. It costs $8.64/h, so it is now the dearest card either endpoint
can be assigned and the video ledger rate rises to $0.0024/s — every video
estimate goes up about 46%, which makes the budget gate more conservative, not
less.
2g-billing. A first-time image pull in a new region is NOT billed#
Read off the endpoint's own Billing panel while a worker had been initializing
for 27 minutes on its first-ever pull in US-CA-2:
No charges in the past 24 hours · Spend rate $0.00000 / second · Total billed $0.00
So the picture is now complete and consistent with V.11.01's account evidence:
RunPod bills execution. Not queue wait, not worker boot, not image pull.
That is why our ledger charging delayTime overstated a clip by 4x, and it is
why a cold region is worth waiting out rather than abandoning — a 27-minute
first pull costs nothing but wall clock.
Check the panel rather than reasoning about it. The endpoint page shows
Spend rate and Total billed live, and it answered in one page-load a
question that is otherwise unanswerable from the API.
2g-probe. US-CA-2 has capacity EU-RO-1 does not (measured 2026-09-12)#
The migration question is no longer speculative. A 10 GB volume in US-CA-2
with a throwaway endpoint carrying the identical nine cards as the video
endpoint was handed a worker inside a minute, with no jobs queued at all,
while EU-RO-1 at the same instant sat throttled with two jobs waiting. Same
card ids, opposite outcome: the constraint is the REGION.
Caveats, because one reading is one reading: the probe ran a tiny runpod/base
image, so it measured card availability and not boot time — a 288 GB volume
and a multi-GB image still have to be pulled wherever we go. US datacenters that
support network volumes: US-CA-2, US-CO-1, US-GA-2, US-IL-1, US-MO-2,
US-NC-2, US-TX-3.
Two traps the probe walked into, so you do not have to.
-
Creating a serverless endpoint through the REST API starts a worker immediately, with no job submitted. RunPod defaults
workersStandbyto 1 (3 whenworkersMaxis left unset), andworkersStandbycannot be set at all through REST — it is rejected as an unknown key on both create and patch. So every endpoint you create begins billing at once.PATCH {"workersMax": 0}does stop it, within about 40 seconds. An earlier note here said it did not and that only deleting the endpoint worked; that was wrong — the probe was deleted ~20 s after the patch, before the worker had wound down. Measured onisis-comfyui-video-us:initializingat t+20 s, no workers at t+40 s.So the recipe is: create → immediately
workersMax: 0→ raise it to 1 only when you are ready to submit. That costs well under a minute of one worker instead of however long you take to notice. -
Both real endpoints carry
workersStandby: 1. They are not billing for it today only because EU-RO-1 cannot place a worker at all. That is starvation suppressing a standing cost, not a safe configuration — re-check it before moving to a region that can actually serve us.
2g. The image endpoint was rebuilt, and the region option if GPUs run out#
Endpoint 6jye7vngg4i685 (was 6l1f3tku7z2zoy, deleted 2026-09-12),
template vfl3iu1dr6, container disk 30 GB.
Why it was rebuilt: container disk. The old endpoint's template allocated 20
GB. When V.01.01 added the six video node packs and their dependencies (cupy,
triton, timm, pycocotools, the controlnet_aux annotators) the image outgrew it,
and from Release #10 onward no worker ever started — placement succeeded,
the container was never created, and the worker sat initializing indefinitely
with completely empty logs. Raising the disk on the old endpoint did not
recover it; a rebuild did, and the first job on the new one ran normally.
If a worker ever hangs in initializing with no log output again, check the
container disk against the image size first. It is the one field neither API
exposes for a git-integration template — GET /v1/templates/<id> 404s and
GraphQL podTemplates omits it — so it is console-only, under Manage → Edit
endpoint.
Region is NOT the usual cause, but it is the fallback. Serverless supply per region, measured in the console's endpoint-create form on 2026-09-12 with exactly one datacenter selected (labels are point-in-time and moved measurably within the same session):
| Region | 24 GB | 24 PRO | 32 PRO | 48 PRO | 80 PRO | 96 PRO | 141 GB |
|---|---|---|---|---|---|---|---|
| EU-RO-1 (current) | Low | Low | Low | — | — | Low | — |
| US-CA-2 | — | Low | — | — | Low | Low | Low |
| US-IL-1 | Low | High | — | Low | — | — | — |
| US-GA-2 | Low | — | — | — | Low | — | Low |
| US-MO-2 | Low | — | — | — | — | Low | — |
| US-TX-3 | — | Low | — | Low | — | — | — |
| US-NC-2 | — | — | — | — | — | Low | — |
| US-CO-1 | — | — | — | — | — | — | Low |
Read it this way: EU-RO-1 is the only region with 32 GB PRO, which matters because Chroma1-HD (17.8 GB) plus T5-XXL (9.8 GB) is a tight fit in 24 GB. US-CA-2 is the only region covering both tiers — image and all three video sizes — so it is the migration target if one is ever needed. US-IL-1 has the only High reading anywhere (24 GB PRO) but nothing for video.
Trigger for migrating (user decision 2026-09-12): only if no GPU available /
sustained throttled starts recurring. It is not worth doing pre-emptively.
The move is: create a 500 GB volume in the target region, run the C.02.06 loader
against the same manifest (~$0.06 of CPU pod, ~1 hour for 358 GB), verify all 42
sha256s, rebuild both endpoints there, and only then delete the EU-RO-1 volume.
Keep both volumes until the new one verifies — the overlap is about $0.05/h.
Two API traps if you do migrate. NVIDIA RTX PRO 4500 Blackwell — the card
that ran every successful C.11 job — is not accepted by the serverless REST
API, and PATCHing it sets gpuTypeIds to null rather than erroring, leaving
the endpoint with no GPUs at all. It is only reachable as the pool id
BLACKWELL_32, which REST rejects, and the console's "32 GB PRO" tier maps to
ADA_32_PRO instead. Also, REST reports a stale gpuTypeIds for an
endpoint whose GPUs were changed through the pool API — trust the console.
2b. Dev credential#
Provider routes (/api/v1/jobs/runpod/*, /ready detail) need a bearer token
with an operator role; the API has no API-key validator wired. Add a throwaway
JWT_SECRET (≥ 32 chars, e.g. openssl rand -hex 32) to the env recipe, then:
TOKEN=$(node scripts/isis/mint-dev-jwt.mjs --hours 12) # HS256, admin role, project_isis_dev
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:4010/api/v1/jobs/runpod/cost/today
Paste the same token into the dashboard's Login page (bearer token field).
3. Preflight (refuses to continue on any incoherence)#
node scripts/isis/dev-stack-preflight.mjs
node scripts/isis/validate-chroma-endpoint-config.mjs --require-image
4. Generation API (port 4010)#
memory_pressure | tail -1
( cd apps/isis/generation-api && pnpm dev ) > /tmp/isis-gen-api.log 2>&1 &
echo $! >> /tmp/claude-<session>.pids
until curl -sf http://127.0.0.1:4010/health >/dev/null; do sleep 2; done
curl -s http://127.0.0.1:4010/health | head -c 400
curl -s http://127.0.0.1:4010/api/v1/jobs/runpod/cost/today
Log lines to expect: Inline job runner started,
Workflow catalog bootstrap complete … includeChromaCatalog: true,
RunPod adapter initialized.
tsx servers do not pick up source edits — restart after any change.
5. Dashboard (Vite, port 5180)#
( cd apps/isis/web && VITE_ISIS_GENERATION_API_URL=http://127.0.0.1:4010 pnpm dev ) \
> /tmp/isis-web.log 2>&1 &
echo $! >> /tmp/claude-<session>.pids
Open http://127.0.0.1:5180 → Login accepts a pasted bearer token or API key;
with NODE_ENV=development the API allows anonymous access.
The Workflows page is backed by the separate workflow-registry service
(user-authored workflows, port 3003), which does not know the catalog, so the
Chroma flows are submitted from Job Submission Studio: type text-to-image
(or image-to-image / inpainting), prompt, and in Extra parameters
(JSON):
{
"workflowId": "chroma-txt2img",
"checkpoint": "chroma1-hd",
"aspect": "square_1024"
}
Every catalog input travels under its own name (checkpoint, aspect, cfg,
steps, seed, denoise, model, text_prompt, grow_mask_by,
confidence_threshold); image inputs (source_image, mask_image) take a
data:image/png;base64,… string. The Jobs page shows the run (claimed by
inline-<pid>) and, once completed, the output thumbnail with seed, checkpoint,
measured cost and execution time; the RunPod page shows spend.
5b. Worker outputs to Hetzner Object Storage (C.12.06 / V.12.05, decided 2026-09-13)#
The stock worker's uploader names its bucket after the month and cannot sign for
a non-AWS host, so the image carries an oshun_uploader.py (imported at
start-up through an oshun_uploader.pth) that replaces rp_upload.upload_image
whenever BUCKET_ENDPOINT_URL is set (see
docker/runpod/comfyui-slim/oshun_uploader.py). Outputs then land in Hetzner
Object Storage and come back to the API as presigned URLs, which the API fetches
into its own output store — so the browser still sees one kind of URL.
Set these on BOTH endpoints in the RunPod console (Manage → Edit endpoint →
Environment variables; a git-integration template's env cannot be edited through
either API). The two secrets come from
~/Desktop/workspace/drawer/env-master.env (HETZNER_APP_DEV_S3_ACCESS_KEY,
HETZNER_APP_DEV_S3_SECRET_KEY); copy the values, never commit them.
| Key | Value |
|---|---|
BUCKET_ENDPOINT_URL |
https://hel1.your-objectstorage.com |
BUCKET_ACCESS_KEY_ID |
the Hetzner access key |
BUCKET_SECRET_ACCESS_KEY |
the Hetzner secret key |
BUCKET_NAME |
app-dev-server |
BUCKET_KEY_PREFIX |
isis/outputs |
BUCKET_REGION |
hel1 |
COMFY_LOG_LEVEL |
INFO (the DEBUG default floods the log) |
COMFY_POLLING_MAX_RETRIES |
1500 (image) / 3600 (video) |
COMFY_POLLING_INTERVAL_MS |
250 |
Objects are keyed isis/outputs/<jobId>/<8 hex><ext>. The API side needs
ISIS_WORKER_OUTPUT_FETCH_HOSTS=hel1.your-objectstorage.com (§2); any other
host in a worker's s3_url is refused with the reason recorded on the output.
The worker log prints oshun uploader: rp_upload.upload_image -> bucket … at
start when the patch is active, which is the check that the env reached it.
5c. Model library and cache (option B, tracker L.00–L.07, decided 2026-09-13)#
Where weights live. The source of truth is the Hetzner library: bucket
app-dev-server on https://hel1.your-objectstorage.com, prefix
isis/models/, key = prefix + the manifest row's destPath
(manifest.library). The RunPod network volume is a cache of it
(manifest.cache, 400 GB, isis-model-cache): workers read from the cache, the
cache is filled from the library. Pinned families (manifest.families[].pinned:
the Chroma stack with its LoRAs, both Wan 2.2 expert pairs, the Wan shared
encoders, SAM 3, BiRefNet, FlashVSR, GIMM-VFI, Real-ESRGAN — 212.26 GB) are
never evicted; every other family stays until space is needed and then the
family used longest ago leaves first. Shared files (umt5, the Wan VAE,
clip_vision_h) leave only when no remaining family needs them, and a family a
queued or running job leases is never evicted.
Adding a model. (1) A manifest row with families (lint
node scripts/isis/runpod-volume-manifest-lint.mjs --strict must stay green,
including the cache invariants it prints); (2) if a workflow uses it, add the
family to that workflow's requires_families (model-families.spec.ts fails
otherwise); (3) put the bytes in the library — library-pull streams from
Hugging Face or Civitai with no local copy:
node scripts/isis/runpod-provision.mjs pod create-loader --manifest <manifest-with-the-new-rows> \
--mode library-pull --dc EU-RO-1 --parallel 8 --dry-run # keys show as <n chars>
node scripts/isis/runpod-provision.mjs pod create-loader ... --yes
(4) the cache picks it up the first time a job needs its family. Nothing is copied to the cache by hand.
Loader modes (LOADER_MODE, scripts/isis/runpod-volume-loader.py):
library-seed (volume → library), library-pull (source → library),
library-verify (download and hash every row, write verifiedAt),
library-probe (L.00.01 transfer measurement), cache-sync (a plan of
evictions and pulls, applied to the mounted cache). Every mode writes
/runpod-volume/.oshun/<mode>-result.json (read it back with
runpod-volume-s3-put.mjs against the volume's S3 API) and prints a RESULT
line; delete the pod afterwards — it sleeps an hour for that purpose and bills
$0.06/h meanwhile. The two Hetzner keys are read from the env names in
manifest.library.auth and injected as pod env; --dry-run redacts them.
Single PUTs up to 512 MiB carry the manifest sha256 as the signed payload hash
(the server refuses wrong bytes); above that, multipart with a per-part hash and
a whole-file hash computed while reading — a mismatch aborts the upload.
Hetzner's multipart works (measured 2026-09-13); RunPod's volume S3 multipart
does not, which is why nothing uploads to the volume over S3 above 4 GiB.
Warming a session. Before a session that will use unpinned families
(LTX-2.3, Animate, VACE, the controlnet_aux preprocessors, RMBG-2.0), warm them:
POST /api/v1/runpod/cache/sync with
{"workflows": ["ltx23-av"], "dryRun": true} shows the pulls, the evictions and
the estimated wait; the same body without dryRun starts the sync (one per
volume at a time; it bills a cache_sync line at the CPU pod rate, and the kill
switch and budget guardrail apply). The dashboard's RunPod page has the same
"Warm for session" picker.
409 models_not_cached. A job whose families are not cached is refused at
submit, before any billing, with the families, their bytes, the estimated wait
and the exact sync call to make. With ISIS_RUNPOD_CACHE_AUTOSYNC=true the job
is held as awaiting_cache and the sync starts for it instead. When no cache
volume is configured (ISIS_RUNPOD_CACHE_VOLUME_ID and
manifest.cache.volumeId both unset) the gating is off and jobs run against the
old full volume.
Pinning or unpinning a family. Flip pinned in manifest.families, run the
lint (the invariants must still hold at the cache size), commit. Pinning does
not pull anything by itself; the next sync that plans for it keeps it.
Licences (A.04.04). Every weight's licence, territory, acceptable-use
clause, Civitai flags, content policy and distribution is in
model-licences.md, generated from the manifest by
node scripts/isis/model-licence-register.mjs (its test fails when the file
drifts from the manifest). Adding a model edits the manifest row's licence
string and regenerates the register in the same commit; the exclusions table
names what must never be added.
Outputs retention and the storage alarm (L.07.02). Worker outputs under
isis/outputs/ share the bucket, and the account's included allowance, with the
library. The API's Hetzner storage guard sweeps outputs older than
ISIS_HETZNER_OUTPUTS_RETENTION_DAYS (7) every
ISIS_HETZNER_STORAGE_GUARD_INTERVAL_MS (6 h), then sums every bucket the
credentials list and raises storage.hetzner.usage_high at
ISIS_HETZNER_USAGE_ALARM_GB (900). The library prefix is refused by the S3
client itself, whatever the env says. Every over-the-line check logs
ops.hetzner_storage.usage_high at error level; the webhook goes out once per
UTC day with event id storage.hetzner.usage_high:<date>. The same two jobs
from a shell:
node scripts/isis/runpod-volume-s3-put.mjs hetzner-usage # exit 2 when over the line
node scripts/isis/runpod-volume-s3-put.mjs hetzner-sweep --days 7 # add --yes to delete
Verifying the library. library-verify over the manifest downloads and
hashes every row. The old 1024 GB volume ax0oj250j3 was deleted on 2026-09-13
at 18:13Z on the user's explicit go (L.06.05), after both endpoints were
repointed to 5bst2pgw7y and its loader result files were copied to
docs/agents/evidence/isis-chroma-runpod/l-results/. The library is now the
only full copy of the weights; a lost cache volume is rebuilt by syncing
families from it.
6. Submitting from the API instead (mask-only flows, scripts)#
curl -s -X POST http://127.0.0.1:4010/api/v1/jobs -H 'Content-Type: application/json' -d '{
"type": "text-to-image",
"prompt": "a red bicycle leaning on a white wall, morning light",
"parameters": { "workflowId": "chroma-txt2img", "checkpoint": "chroma1-hd", "aspect": "square_1024", "seed": 42 }
}'
curl -s http://127.0.0.1:4010/api/v1/jobs/<jobId>
A refusal with RUNPOD_DAILY_BUDGET_EXCEEDED (429) or
RUNPOD_COST_LEDGER_UNAVAILABLE (503) is the cost gate working, not a bug.
To run one catalog workflow straight at the endpoint, bypassing the API (this is
what the C.11 matrix used), it is npx tsx, not node — the script imports
catalog-renderer.ts at runtime and that module's extensionless imports are
unresolvable to plain Node ESM, which fails as
Cannot find module .../expression-evaluator:
npx tsx scripts/isis/runpod-catalog-run.mjs matting-remove-background \
--image source_image=docs/agents/evidence/isis-chroma-runpod/chroma-txt2img-0-9d49c5b4.png \
--set model=rmbg-2.0 --out /tmp/run-out
It needs RUNPOD_API_KEY and RUNPOD_COMFYUI_ENDPOINT_ID only — no dev stack.
It gives up after 15 minutes in IN_QUEUE, which on this endpoint is now a real
possibility: see §2e.
6c. Getting a failed job's worker log (it is a stream, not a document)#
Three failures on 2026-09-12 — wan22-animate killing ComfyUI,
flashvsr-upscale returning a bare Job processing failed, and ltx23-av
before its cause was found in the graph — were all opaque for the same reason,
and on 2026-09-13 that reason turned out to be a bug in the capture code,
not the platform.
GET https://api.runpod.io/v2/serverless/<endpoint>/workers/<workerId>/logs
answers content-type: text/event-stream: id: plus data: frames, one JSON
object each ({"source":"system"|"container","line":…,"ts":…}), replayed from
the worker's first line at roughly eight frames a second and then held open for
live lines. It never closes. Both harnesses did await response.text() on it,
which can never resolve, and their 20 s timeout then wrote "log fetch timed out"
into every failure row. Measured live: ?tail=N is honoured (the last N lines,
then live); Last-Event-ID and ?since= are not; once the worker is recycled
the URL answers 404 "worker not found" in about three seconds — that part of
the old explanation was right, and it still means a log is a race you can lose.
The reader is scripts/isis/runpod-worker-log.mjs: it asks for the tail,
collects frames until the stream goes quiet for 2.5 s or a 20 s cap lands, and
writes plain <ts> <source> <line> lines. A capped read is marked truncated
rather than passed off as complete. Both runpod-catalog-run.mjs and
motion-live-proof.mjs call it the moment a job comes back FAILED.
node scripts/isis/runpod-worker-log.mjs --endpoint "$RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID" --worker <workerId> --tail 400
node scripts/isis/runpod-worker-log.mjs --endpoint "$RUNPOD_COMFYUI_VIDEO_ENDPOINT_ID" --worker <workerId> --out docs/agents/evidence/isis-chroma-runpod/<run>/worker-<workerId>.log
The worker id is on the job's status response (workerId) and on
GET /v2/serverless/<endpoint>/workers. status.output.errors[] remains the
first place to look — it carried ltx23-av's
Node 8 (LTXVImgToVideoInplace): required_input_missing 'image' with no log at
all — but a Job processing failed with nothing in errors[] is exactly the
case the log exists for.
6b. Reading a RunPod build log (when a git-integration build fails)#
The builds API tells you a build failed and almost nothing else:
curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \
"https://api.runpod.io/v2/serverless/$RUNPOD_COMFYUI_ENDPOINT_ID/builds" | jq '.[0]'
# -> "Command '['/usr/bin/docker', 'buildx', 'build', ...]' returned non-zero exit status 1."
There is no /logs route under it — every shape 404s. Use:
node scripts/isis/runpod-build-log.mjs --latest # the most recent build
node scripts/isis/runpod-build-log.mjs <buildId> # a specific one
node scripts/isis/runpod-build-log.mjs --latest | grep -nE "ERROR|IMPORT FAILED|Traceback|error:"
It calls the same getBuildLogs GraphQL operation the console uses, with the
ordinary API key, and reverses the result — the API returns newest-first, and a
build log reads oldest-first. The first real error is near the top; the generic
docker summary is at the end and is never the reason.
Do not skip this and change the Dockerfile on a hypothesis. Two builds were
spent that way on 2026-09-12: the guess was import cupy on a GPU-less builder,
the reality was a dangling symlink breaking ComfyUI-SAM3's os.makedirs, and
the log said so plainly the whole time. It is the same lesson as C.12.12.
If you try the console instead: a RunPod console screenshot can come back
completely blank while the page is fully rendered and interactive. Check
document.body.innerText.length before concluding the page is broken. The log
panel is also virtualised, so scraping the DOM only ever gets the ~15 lines
currently on screen.
7. Teardown (every session)#
for p in $(cat /tmp/claude-<session>.pids); do kill $p 2>/dev/null; done
lsof -nP -iTCP -sTCP:LISTEN | grep -E ':(4010|5180)' # must be empty
rm -f /tmp/isis-chroma-env.sh
node scripts/isis/runpod-provision.mjs account # pods 0, spend/h = volume only