Eve library

ADR-0010: A Hetzner Model Library with a Fixed Least-Recently-Used Cache Volume on RunPod

ADR

Revision 1

On this page

Status: Accepted
Date: 2026-09-13 (decided), 2026-09-18 (recorded)
Owner: @GreyChimp (the decision and every live step are the owner's)
Authors: Isis Runtime Engineering
Related ADRs: ADR-0003 (pay-per-second serverless strategy), ADR-0004 (endpoint lifecycle), ADR-0005 slim worker (no weights in the image; weights on a network volume) — this ADR supersedes that ADR's "one network volume holds every weight" arrangement and keeps everything else in it
Decision record: the owner's decision block dated 2026-09-13 in ISIS_CHROMA_RUNPOD_MVP_TODOS_2026-09-11.md ("Decision taken by the user on 2026-09-13 (model library on Hetzner, 400 GB LRU cache on RunPod)"), planned in commit c2d118585f7
Evidence: docs/agents/isis-chroma-runpod-evidence.md §§ L.00–L.07 (every pod, its window, its wall clock, its cost and its result file)
Runbook: docs/domains/isis/runbooks/chroma-runpod-dev-stack.md § 5c

1. Context and Problem Statement#

ADR-0005 (slim worker) put every weight on a RunPod network volume and kept images free of model files. That held while the manifest was 50 rows and 367 GB. Two things then changed at once.

The arsenal expansion added the survey models to scope, taking the manifest past 700 GB. Under the volume lint's 20 % headroom rule that sized a volume at 1024 GB, which RunPod prices in tiers at $71.20 a month.

The spend profile made that the wrong shape. A RunPod network volume bills every hour it exists, whether or not a worker touches it. Generation in this project happens in sessions, at certain hours. The whole eight-clip V.11 proof matrix cost roughly $3 of GPU (evidence § V.11); storage had become ten to twenty times the compute spend, for bytes that sit idle most of the week.

Meanwhile the account already pays for Hetzner Object Storage for another purpose. Its base price includes 1 TB of storage and 1 TB of egress per account, ingress is free, and egress past the allowance is €1.00/TB. RunPod charges nothing for data in. So the same 700 GB of weights costs approximately nothing extra on Hetzner, and moving them there converts a standing monthly charge into a per-miss wait, not a per-miss cost.

The problem to decide: where the weights live, what the RunPod side becomes, how big it is, and what leaves it when it is full.

2. Decision#

The Hetzner bucket is the library and the source of truth for weights. The RunPod network volume becomes a fixed 400 GB cache of it, holding a pinned core plus whatever was used most recently, evicted least-recently-used first and only when an incoming family does not fit.

Concretely:

  1. Library. manifest.library in infra/runpod/volumes/isis-chroma-models.manifest.json: provider: hetzner-object-storage, endpoint https://hel1.your-objectstorage.com, region hel1, bucket app-dev-server, prefix isis/models/, credentials named (never valued) as HETZNER_APP_DEV_S3_ACCESS_KEY and HETZNER_APP_DEV_S3_SECRET_KEY. A row's library key is prefix + destPath — the same string as its path under the cache's modelsRoot, so there is one name for a file everywhere.
  2. Cache. manifest.cache: isis-model-cache, volume 5bst2pgw7y, 400 GB, EU-RO-1, marginRatio 0.05, invariantRatio 0.95. Workers read from the cache; the cache is filled from the library; nothing is copied to the cache by hand.
  3. The unit of residency is a family, not a file. manifest.families gives each family an id, a pinned flag and a tier; each artifact lists every family that needs it, so a shared file belongs to several. Each catalog workflow declares requires_families, and the static validator fails with MODEL_NOT_IN_FAMILIES when a rendered graph references a model outside the declared families — including inside _condition branches and enum values — so the declaration is checked rather than trusted.
  4. Admission at submit, before anything is billed. A job whose families are not all cached is refused with 409 models_not_cached carrying the families, their bytes, an estimated wait and the exact sync call to make; with ISIS_RUNPOD_CACHE_AUTOSYNC=true it is instead held as awaiting_cache and a sync is started or joined for it.
  5. Eviction is planned by a pure function (planCacheSync) and applied by a CPU loader pod, never by the GPU worker.
  6. The old 1024 GB volume is deleted once the library is download-verified and the cache is proven, not before.

3. Options Weighed#

Option Monthly storage Outcome
A. Keep the 1024 GB volume, every weight resident $71.20 Rejected. Pays peak size every hour of every week for bytes touched in a few sessions.
B. Hetzner library plus a fixed cache volume $28.00 Chosen. Trades a standing charge for a first-use wait, which the miss log can then shrink by pinning.
C. Create and destroy a volume per session ~$0 idle Rejected. Both endpoints would need re-pointing every session — a path that has wedged an endpoint before — and every session pays a full pull of everything.
D. No volume; pull into each worker's container disk $0 Rejected. Every cold worker re-pulls while GPU-billed; the wait moves from free CPU-pod time onto the expensive meter.

Option D is the one worth keeping in mind, because it is what a naive reading of "serverless" suggests. The measured cost of a miss under option B is CPU-pod time at $0.06/hour plus wall clock; under option D the same bytes arrive on a GPU worker at two orders of magnitude more per second.

4. The Size, and Why It Is 400 GB#

Sized by the invariant, not by a headroom percentage: the pinned core plus the largest single unpinned family must fit inside invariantRatio × cache size, and every family must fit beside the core — a family that cannot fit beside the pinned core can never load at all, which is a configuration bug the lint catches rather than a runtime surprise.

At the decision (2026-09-13): pinned core 212.26 GB, largest unpinned family Qwen-Image 106.32 GB, total 318.58 GB of the 380 GB allowed at 400 GB.

Live today (node scripts/isis/runpod-volume-manifest-lint.mjs --strict, 99 artifacts, library 700.39 GB):

text
cache 400 GB: pinned 239.03 GB + largest unpinned ltx25 92.20 GB = 331.22 GB of 380 GB allowed

Sizes rejected and why: 200 GB cannot hold the core; 300 GB holds the core but can never load Qwen-Image beside it; 450 GB would be needed only for the one pairing of Qwen-Image and LTX on the same day. RunPod volumes only grow, never shrink, so the rule is to start at 400 and grow on the evidence of the miss log, never on a guess.

The pinned core is the set that must never make a member wait: at the decision, the Chroma stack with its four LoRAs, both Wan 2.2 expert pairs, the Wan shared encoders, SAM 3, BiRefNet, FlashVSR, GIMM-VFI and Real-ESRGAN — 212.26 GB. It is not frozen: pinning is a pinned flag in the manifest plus a green lint, and the set has already moved (SAM 3 → SAM 3.1, Real-ESRGAN out, SeedVR2 3B in). Pinning pulls nothing by itself; the next sync that plans for the family keeps it.

5. The Eviction Rules#

The rules are implemented in apps/isis/generation-api/src/services/runpod-model-cache/planner.ts as a pure function over its input — no I/O — so they can be read and tested in one place. planCacheSync returns a hit, a plan of pulls and evictions, or a typed refusal, and a refusal moves no bytes.

  1. The core is never evicted, and neither is anything in use. A pinned family, a family leased by a queued or running job, and the family being requested are all protected. Deleting weights under a live ComfyUI crashes it, and a crashed worker poisons every job after it — hence leases.
  2. Files are reference-counted; the eviction unit is the family. A file leaves only when no family that remains cached, syncing, pinned, leased or requested still references it. This is not a detail: umt5 serves Wan, Animate-2 and SCAIL-2; ae.safetensors serves Chroma and Z-Image; the Qwen encoder serves Z-Image and klein 4B. Evicting a family by deleting "its" files would silently break three others.
  3. Eviction only on demand. Nothing is evicted when used + bytesToPull + marginRatio × cacheBytes already fits. Otherwise candidates are evicted one at a time until the request fits, and no further — a sync never tidies up.
  4. Least recently used first. A family synced but never used counts from its syncedAt; ties break by family id, so a plan is deterministic.
  5. "Last used" is written when a job COMPLETES, from the job ledger. A failed job does not refresh it, so a family that only ever fails ages out; filesystem access times are never consulted.
  6. If it cannot fit, it fails before any byte moves. Four typed refusals, each naming what stands in the way: exceeds_cache (the pinned core plus the request can never fit at this size), blocked_by_leases (it would fit once running jobs release), blocked_by_volume_state (unmanaged bytes or families still syncing occupy the space), unknown_family.

Two cases the decision did not spell out, settled in the implementation and recorded here because they are the ones that bite in operation:

  • Residue. A sync that fails part-way leaves files on disk under an absent family, and a crashed eviction leaves a family evicting. Those bytes are real, so they count as used, and such a family is an eviction candidate ahead of healthy cached families — residue serves no job. Ordering is evicting, then absent, then cached, LRU inside each group. A syncing family is never a candidate: the sync executor owns it.
  • Unmanaged bytes. Objects under models/ that are not a manifest file at its manifest size — short copies, .partial leftovers, retired weights — are measured by the reconciler, count as used, and are never freed by a plan. When they are what stands in the way, the refusal is blocked_by_volume_state and says so.

The table is never trusted on its own. Before every plan, the reconciler lists the volume through its S3 API; a family is cached only when every one of its files is present at the manifest size, and a recorded cached family with a missing or short file flips to absent with a logged event.

State lives in three tables plus the sync lock: runpod_cache_families (state absent | syncing | cached | evicting, bytes, last_used_at, synced_at), runpod_cache_leases (family, job, acquired at), append-only runpod_cache_events (hit, miss, sync started, sync done, sync failed, evicted — a trigger blocks UPDATE, DELETE and TRUNCATE), and runpod_cache_syncs with a partial unique index allowing one running sync per volume. Leases are acquired at submit in the same transaction as the cached check, which is what makes the race between a submit and an eviction of the same family decidable; it is tested against real Postgres in both forced orders and in 25 concurrent rounds with exactly one winner each.

6. How Bytes Move#

One loader program, scripts/isis/runpod-volume-loader.py (stdlib only, including its own SigV4 signer checked against eight botocore vectors). Option B added five modes beside its original volume-seed:

Mode Direction Purpose
library-seed volume → library The one-time migration of what was already on the volume.
library-pull source → library Hugging Face or Civitai straight into the library, no local copy beyond one part buffer.
library-verify library → /dev/null Download and hash every row; writes verifiedAt.
library-probe both The transfer measurement of L.00.01.
cache-sync library → cache Applies a plan of evictions and pulls to the mounted cache.

Integrity is checked on every leg, because the storage will not do it for us. The L.00.01 probe established that Hetzner does not validate a full-object x-amz-checksum-sha256 on multipart completion — a deliberately wrong checksum was accepted — so download-side hashing stayed in full scope for every row. Single PUTs up to 512 MiB carry the manifest sha256 as the signed payload hash, so the server itself refuses wrong bytes; above that, multipart with a per-part hash and a whole-file hash computed while reading, and a mismatch aborts the upload rather than leaving a bad object. A cache-sync pull lands through .partial, is verified, and only then renamed into place.

The sync orchestrator takes the volume's lock, reconciles, plans, marks the planned evictions evicting and the pulls syncing, creates the CPU loader pod, then in the background polls the result file, applies it, deletes the pod however the sync ended, books the pod's wall-clock life as a cache_sync line in the daily ledger, releases the lock and settles jobs held for the cache. A sync that cannot start puts the families back. A pull family is cached only when the loader reported it cached; anything else leaves it absent with a sync_failed event.

7. Consequences#

Cost. Storage falls from $71.20 to $28.00 a month; with the old volume gone the cache alone is $0.039/hour. During the migration both volumes existed at about $0.171/hour, which is why L.06.05's deletion was not left to wait.

A miss now costs time. A first use of an unpinned family waits for a sync. Measured end to end on 2026-09-13 (evidence § L.06.04): submitting wan21-vace-edit was refused at 19:10:25Z with 409 models_not_cached naming controlnet-aux (1.69 GB) and wan21-vace (46.30 GB, of which 36.37 GB to pull because umt5 was already cached with wan-shared); the sync finished 24.4 minutes later; the clip rendered after about 21 minutes of GPU queue and 5.7 of render. Miss to render: 52 minutes. Warming before a session is therefore part of operation, not an optimisation — POST /api/v1/runpod/cache/sync with {"workflows": [...], "dryRun": true} shows the pulls, the evictions and the estimated wait, and the dashboard's RunPod page has the same picker.

Every wait estimate is measured or absent. The estimator computes from ISIS_RUNPOD_CACHE_SYNC_MB_PER_SECOND, which the runbook sets to the probe's measured 20.8 MiB/s; unset, the 409 body and the dry-run plan return null with a reason rather than a guess.

The library is now the only full copy. ax0oj250j3 (1024 GB) was deleted at 18:13:21Z on 2026-09-13 on the owner's explicit go, after both endpoints were repointed to 5bst2pgw7y, every row had been download-verified (L.01.05) and the loader result files had been copied into docs/agents/evidence/isis-chroma-runpod/l-results/. A lost cache volume is rebuilt by syncing families from the library. A lost library is not rebuildable from the cache.

The Hetzner allowance is shared, per account. Worker outputs under isis/outputs/ draw on the same 1 TB as isis/models/. The API's storage guard sweeps outputs older than ISIS_HETZNER_OUTPUTS_RETENTION_DAYS (7) every ISIS_HETZNER_STORAGE_GUARD_INTERVAL_MS (6 h), sums every bucket the credentials list, and raises storage.hetzner.usage_high at ISIS_HETZNER_USAGE_ALARM_GB (900), once per UTC day. The sweepable-prefix allowlist lives in the S3 client (HETZNER_SWEEPABLE_PREFIXES), not in the guard's env, so the guard cannot be configured toward the library — the sweeper cannot delete weights. The manifest lint additionally warns past 1000 GB of library.

Adding a model is four steps, none of them a copy. A manifest row with its families (lint green, including the cache invariants it prints); the family added to any workflow's requires_families (model-families.spec.ts fails otherwise); the bytes into the library with library-pull; and the cache picks it up the first time a job needs that family.

Retirement means the library and the cache. Retiring a model deletes it from the library and evicts it from the cache; it no longer means deleting from a volume. authoring/generation-retirements.json in the presentation center reads these deletions from the commits that made them.

Transfers are not reliable and the tooling assumes it. The seed of 50 rows took four launches: two pods stalled on S3 connections Hetzner stopped answering, one hung 40 minutes at 0 % CPU on a volume read. The fixes are in the loader — a 30 s stall timeout (a94f2832105), a watchdog that exits 75 with a start loop that reruns (111a49e7518), a partial-file resume instead of a restart (e0c6d74442e), and a source-connection resume by range request (d78f80df034). The successful run was 2478 s for 367.17 GB, and a listing confirmed 50 objects with every size and every x-amz-meta-sha256 matching the manifest. Contention is real: a cache sync and a library pull against the same bucket at the same time stalled the sync.

One defect this design produced, found in use and fixed. A sync driver refreshed its heartbeat only inside its result-polling loop, so during a long applying phase the heartbeat went stale, resumeStaleSync() took the sync over inside the same process, and the pod's cost was booked twice (2026-09-14, sync b80de934). The driver now heartbeats on a timer in every phase and resumeStaleSync() never takes over a sync this process is still driving; both specs were shown to fail with the fix removed (5ceea7b97c9).

8. Status of the Implementation#

Delivered under tracker sections L.00–L.07 and recorded item by item in docs/agents/isis-chroma-runpod-evidence.md. The service is apps/isis/generation-api/src/services/runpod-model-cache/ (manifest, planner, reconciler, store, sync service, admission, dashboard projection), with the store and race specs run against real Postgres and every outside boundary — volume listing, result file, pod API, spend gate, ledger, job service — injected, so the orchestration is tested with the real planner and the real store and nothing else doubled.