Status: Accepted
Date: 2026-09-13 (decided), 2026-09-18 (recorded)
Owner: @GreyChimp (the decision and every live step are the owner's)
Authors: Isis Runtime Engineering
Related ADRs: ADR-0003
(pay-per-second serverless strategy),
ADR-0004 (endpoint
lifecycle),
ADR-0005 slim worker (no
weights in the image; weights on a network volume) — this ADR supersedes that
ADR's "one network volume holds every weight" arrangement and keeps everything
else in it
Decision record: the owner's decision block dated 2026-09-13 in
ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md ("Decision taken by the user on
2026-09-13 (model library on Hetzner, 400 GB LRU cache on RunPod)"), planned in
commit c2d118585f7
Evidence: docs/agents/isis-chroma-runpod-evidence.md §§ L.00–L.07 (every
pod, its window, its wall clock, its cost and its result file)
Runbook: docs/domains/isis/runbooks/chroma-runpod-dev-stack.md § 5c
1. Context and Problem Statement#
ADR-0005 (slim worker) put every weight on a RunPod network volume and kept images free of model files. That held while the manifest was 50 rows and 367 GB. Two things then changed at once.
The arsenal expansion added the survey models to scope, taking the manifest past 700 GB. Under the volume lint's 20 % headroom rule that sized a volume at 1024 GB, which RunPod prices in tiers at $71.20 a month.
The spend profile made that the wrong shape. A RunPod network volume bills every hour it exists, whether or not a worker touches it. Generation in this project happens in sessions, at certain hours. The whole eight-clip V.11 proof matrix cost roughly $3 of GPU (evidence § V.11); storage had become ten to twenty times the compute spend, for bytes that sit idle most of the week.
Meanwhile the account already pays for Hetzner Object Storage for another purpose. Its base price includes 1 TB of storage and 1 TB of egress per account, ingress is free, and egress past the allowance is €1.00/TB. RunPod charges nothing for data in. So the same 700 GB of weights costs approximately nothing extra on Hetzner, and moving them there converts a standing monthly charge into a per-miss wait, not a per-miss cost.
The problem to decide: where the weights live, what the RunPod side becomes, how big it is, and what leaves it when it is full.
2. Decision#
The Hetzner bucket is the library and the source of truth for weights. The RunPod network volume becomes a fixed 400 GB cache of it, holding a pinned core plus whatever was used most recently, evicted least-recently-used first and only when an incoming family does not fit.
Concretely:
- Library.
manifest.libraryininfra/runpod/volumes/isis-chroma-models.manifest.json:provider: hetzner-object-storage, endpointhttps://hel1.your-objectstorage.com, regionhel1, bucketapp-dev-server, prefixisis/models/, credentials named (never valued) asHETZNER_APP_DEV_S3_ACCESS_KEYandHETZNER_APP_DEV_S3_SECRET_KEY. A row's library key isprefix + destPath— the same string as its path under the cache'smodelsRoot, so there is one name for a file everywhere. - Cache.
manifest.cache:isis-model-cache, volume5bst2pgw7y, 400 GB, EU-RO-1,marginRatio0.05,invariantRatio0.95. Workers read from the cache; the cache is filled from the library; nothing is copied to the cache by hand. - The unit of residency is a family, not a file.
manifest.familiesgives each family an id, apinnedflag and a tier; each artifact lists every family that needs it, so a shared file belongs to several. Each catalog workflow declaresrequires_families, and the static validator fails withMODEL_NOT_IN_FAMILIESwhen a rendered graph references a model outside the declared families — including inside_conditionbranches and enum values — so the declaration is checked rather than trusted. - Admission at submit, before anything is billed. A job whose families are
not all cached is refused with
409 models_not_cachedcarrying the families, their bytes, an estimated wait and the exact sync call to make; withISIS_RUNPOD_CACHE_AUTOSYNC=trueit is instead held asawaiting_cacheand a sync is started or joined for it. - Eviction is planned by a pure function (
planCacheSync) and applied by a CPU loader pod, never by the GPU worker. - The old 1024 GB volume is deleted once the library is download-verified and the cache is proven, not before.
3. Options Weighed#
| Option | Monthly storage | Outcome |
|---|---|---|
| A. Keep the 1024 GB volume, every weight resident | $71.20 | Rejected. Pays peak size every hour of every week for bytes touched in a few sessions. |
| B. Hetzner library plus a fixed cache volume | $28.00 | Chosen. Trades a standing charge for a first-use wait, which the miss log can then shrink by pinning. |
| C. Create and destroy a volume per session | ~$0 idle | Rejected. Both endpoints would need re-pointing every session — a path that has wedged an endpoint before — and every session pays a full pull of everything. |
| D. No volume; pull into each worker's container disk | $0 | Rejected. Every cold worker re-pulls while GPU-billed; the wait moves from free CPU-pod time onto the expensive meter. |
Option D is the one worth keeping in mind, because it is what a naive reading of "serverless" suggests. The measured cost of a miss under option B is CPU-pod time at $0.06/hour plus wall clock; under option D the same bytes arrive on a GPU worker at two orders of magnitude more per second.
4. The Size, and Why It Is 400 GB#
Sized by the invariant, not by a headroom percentage: the pinned core plus the
largest single unpinned family must fit inside invariantRatio × cache size,
and every family must fit beside the core — a family that cannot fit beside
the pinned core can never load at all, which is a configuration bug the lint
catches rather than a runtime surprise.
At the decision (2026-09-13): pinned core 212.26 GB, largest unpinned family Qwen-Image 106.32 GB, total 318.58 GB of the 380 GB allowed at 400 GB.
Live today (node scripts/isis/runpod-volume-manifest-lint.mjs --strict, 99
artifacts, library 700.39 GB):
cache 400 GB: pinned 239.03 GB + largest unpinned ltx25 92.20 GB = 331.22 GB of 380 GB allowed
Sizes rejected and why: 200 GB cannot hold the core; 300 GB holds the core but can never load Qwen-Image beside it; 450 GB would be needed only for the one pairing of Qwen-Image and LTX on the same day. RunPod volumes only grow, never shrink, so the rule is to start at 400 and grow on the evidence of the miss log, never on a guess.
The pinned core is the set that must never make a member wait: at the decision,
the Chroma stack with its four LoRAs, both Wan 2.2 expert pairs, the Wan shared
encoders, SAM 3, BiRefNet, FlashVSR, GIMM-VFI and Real-ESRGAN — 212.26 GB. It is
not frozen: pinning is a pinned flag in the manifest plus a green lint, and
the set has already moved (SAM 3 → SAM 3.1, Real-ESRGAN out, SeedVR2 3B in).
Pinning pulls nothing by itself; the next sync that plans for the family keeps
it.
5. The Eviction Rules#
The rules are implemented in
apps/isis/generation-api/src/services/runpod-model-cache/planner.ts as a pure
function over its input — no I/O — so they can be read and tested in one place.
planCacheSync returns a hit, a plan of pulls and evictions, or a
typed refusal, and a refusal moves no bytes.
- The core is never evicted, and neither is anything in use. A pinned family, a family leased by a queued or running job, and the family being requested are all protected. Deleting weights under a live ComfyUI crashes it, and a crashed worker poisons every job after it — hence leases.
- Files are reference-counted; the eviction unit is the family. A file
leaves only when no family that remains cached, syncing, pinned, leased or
requested still references it. This is not a detail:
umt5serves Wan, Animate-2 and SCAIL-2;ae.safetensorsserves Chroma and Z-Image; the Qwen encoder serves Z-Image and klein 4B. Evicting a family by deleting "its" files would silently break three others. - Eviction only on demand. Nothing is evicted when
used + bytesToPull + marginRatio × cacheBytesalready fits. Otherwise candidates are evicted one at a time until the request fits, and no further — a sync never tidies up. - Least recently used first. A family synced but never used counts from its
syncedAt; ties break by family id, so a plan is deterministic. - "Last used" is written when a job COMPLETES, from the job ledger. A failed job does not refresh it, so a family that only ever fails ages out; filesystem access times are never consulted.
- If it cannot fit, it fails before any byte moves. Four typed refusals,
each naming what stands in the way:
exceeds_cache(the pinned core plus the request can never fit at this size),blocked_by_leases(it would fit once running jobs release),blocked_by_volume_state(unmanaged bytes or families stillsyncingoccupy the space),unknown_family.
Two cases the decision did not spell out, settled in the implementation and recorded here because they are the ones that bite in operation:
- Residue. A sync that fails part-way leaves files on disk under an
absentfamily, and a crashed eviction leaves a familyevicting. Those bytes are real, so they count as used, and such a family is an eviction candidate ahead of healthycachedfamilies — residue serves no job. Ordering isevicting, thenabsent, thencached, LRU inside each group. Asyncingfamily is never a candidate: the sync executor owns it. - Unmanaged bytes. Objects under
models/that are not a manifest file at its manifest size — short copies,.partialleftovers, retired weights — are measured by the reconciler, count as used, and are never freed by a plan. When they are what stands in the way, the refusal isblocked_by_volume_stateand says so.
The table is never trusted on its own. Before every plan, the reconciler
lists the volume through its S3 API; a family is cached only when every one of
its files is present at the manifest size, and a recorded cached family with a
missing or short file flips to absent with a logged event.
State lives in three tables plus the sync lock: runpod_cache_families (state
absent | syncing | cached | evicting, bytes, last_used_at, synced_at),
runpod_cache_leases (family, job, acquired at), append-only
runpod_cache_events (hit, miss, sync started, sync done, sync failed, evicted
— a trigger blocks UPDATE, DELETE and TRUNCATE), and runpod_cache_syncs with a
partial unique index allowing one running sync per volume. Leases are
acquired at submit in the same transaction as the cached check, which is what
makes the race between a submit and an eviction of the same family decidable; it
is tested against real Postgres in both forced orders and in 25 concurrent
rounds with exactly one winner each.
6. How Bytes Move#
One loader program, scripts/isis/runpod-volume-loader.py (stdlib only,
including its own SigV4 signer checked against eight botocore vectors). Option B
added five modes beside its original volume-seed:
| Mode | Direction | Purpose |
|---|---|---|
library-seed |
volume → library | The one-time migration of what was already on the volume. |
library-pull |
source → library | Hugging Face or Civitai straight into the library, no local copy beyond one part buffer. |
library-verify |
library → /dev/null | Download and hash every row; writes verifiedAt. |
library-probe |
both | The transfer measurement of L.00.01. |
cache-sync |
library → cache | Applies a plan of evictions and pulls to the mounted cache. |
Integrity is checked on every leg, because the storage will not do it for us.
The L.00.01 probe established that Hetzner does not validate a full-object
x-amz-checksum-sha256 on multipart completion — a deliberately wrong
checksum was accepted — so download-side hashing stayed in full scope for every
row. Single PUTs up to 512 MiB carry the manifest sha256 as the signed payload
hash, so the server itself refuses wrong bytes; above that, multipart with a
per-part hash and a whole-file hash computed while reading, and a mismatch
aborts the upload rather than leaving a bad object. A cache-sync pull lands
through .partial, is verified, and only then renamed into place.
The sync orchestrator takes the volume's lock, reconciles, plans, marks the
planned evictions evicting and the pulls syncing, creates the CPU loader
pod, then in the background polls the result file, applies it, deletes the pod
however the sync ended, books the pod's wall-clock life as a cache_sync line
in the daily ledger, releases the lock and settles jobs held for the cache. A
sync that cannot start puts the families back. A pull family is cached only
when the loader reported it cached; anything else leaves it absent with a
sync_failed event.
7. Consequences#
Cost. Storage falls from $71.20 to $28.00 a month; with the old volume gone the cache alone is $0.039/hour. During the migration both volumes existed at about $0.171/hour, which is why L.06.05's deletion was not left to wait.
A miss now costs time. A first use of an unpinned family waits for a sync.
Measured end to end on 2026-09-13 (evidence § L.06.04): submitting
wan21-vace-edit was refused at 19:10:25Z with 409 models_not_cached naming
controlnet-aux (1.69 GB) and wan21-vace (46.30 GB, of which 36.37 GB to pull
because umt5 was already cached with wan-shared); the sync finished 24.4
minutes later; the clip rendered after about 21 minutes of GPU queue and 5.7 of
render. Miss to render: 52 minutes. Warming before a session is therefore
part of operation, not an optimisation — POST /api/v1/runpod/cache/sync with
{"workflows": [...], "dryRun": true} shows the pulls, the evictions and the
estimated wait, and the dashboard's RunPod page has the same picker.
Every wait estimate is measured or absent. The estimator computes from
ISIS_RUNPOD_CACHE_SYNC_MB_PER_SECOND, which the runbook sets to the probe's
measured 20.8 MiB/s; unset, the 409 body and the dry-run plan return null with
a reason rather than a guess.
The library is now the only full copy. ax0oj250j3 (1024 GB) was deleted at
18:13:21Z on 2026-09-13 on the owner's explicit go, after both endpoints were
repointed to 5bst2pgw7y, every row had been download-verified (L.01.05) and
the loader result files had been copied into
docs/agents/evidence/isis-chroma-runpod/l-results/. A lost cache volume is
rebuilt by syncing families from the library. A lost library is not rebuildable
from the cache.
The Hetzner allowance is shared, per account. Worker outputs under
isis/outputs/ draw on the same 1 TB as isis/models/. The API's storage guard
sweeps outputs older than ISIS_HETZNER_OUTPUTS_RETENTION_DAYS (7) every
ISIS_HETZNER_STORAGE_GUARD_INTERVAL_MS (6 h), sums every bucket the
credentials list, and raises storage.hetzner.usage_high at
ISIS_HETZNER_USAGE_ALARM_GB (900), once per UTC day. The sweepable-prefix
allowlist lives in the S3 client (HETZNER_SWEEPABLE_PREFIXES), not in the
guard's env, so the guard cannot be configured toward the library — the sweeper
cannot delete weights. The manifest lint additionally warns past 1000 GB of
library.
Adding a model is four steps, none of them a copy. A manifest row with its
families (lint green, including the cache invariants it prints); the family
added to any workflow's requires_families (model-families.spec.ts fails
otherwise); the bytes into the library with library-pull; and the cache picks
it up the first time a job needs that family.
Retirement means the library and the cache. Retiring a model deletes it from
the library and evicts it from the cache; it no longer means deleting from a
volume. authoring/generation-retirements.json in the presentation center reads
these deletions from the commits that made them.
Transfers are not reliable and the tooling assumes it. The seed of 50 rows
took four launches: two pods stalled on S3 connections Hetzner stopped
answering, one hung 40 minutes at 0 % CPU on a volume read. The fixes are in the
loader — a 30 s stall timeout (a94f2832105), a watchdog that exits 75 with a
start loop that reruns (111a49e7518), a partial-file resume instead of a
restart (e0c6d74442e), and a source-connection resume by range request
(d78f80df034). The successful run was 2478 s for 367.17 GB, and a listing
confirmed 50 objects with every size and every x-amz-meta-sha256 matching the
manifest. Contention is real: a cache sync and a library pull against the same
bucket at the same time stalled the sync.
One defect this design produced, found in use and fixed. A sync driver
refreshed its heartbeat only inside its result-polling loop, so during a long
applying phase the heartbeat went stale, resumeStaleSync() took the sync
over inside the same process, and the pod's cost was booked twice
(2026-09-14, sync b80de934). The driver now heartbeats on a timer in every
phase and resumeStaleSync() never takes over a sync this process is still
driving; both specs were shown to fail with the fix removed (5ceea7b97c9).
8. Status of the Implementation#
Delivered under tracker sections L.00–L.07 and recorded item by item in
docs/agents/isis-chroma-runpod-evidence.md. The service is
apps/isis/generation-api/src/services/runpod-model-cache/ (manifest, planner,
reconciler, store, sync service, admission, dashboard projection), with the
store and race specs run against real Postgres and every outside boundary —
volume listing, result file, pod API, spend gate, ledger, job service —
injected, so the orchestration is tested with the real planner and the real
store and nothing else doubled.