How Isis will decide, per lane, whether to keep generating on its own GPUs or send that kind of work to an API — and what part of that machinery exists today.
The decision it serves is the owner's, taken on 14 September 2026:
self-hosting versus APIs is decided per lane by measurement, not all or
nothing. adult_ok lanes stay self-hosted by construction, because hosted
services refuse nudity.
Code: libs/isis/workflows/src/benchmarks/benchmark-suite.ts (the suite),
benchmark-rubric.ts (the rubric, the score record and the aggregate), and
their two spec files. Tracker: E.01.01–E.01.07. Evidence:
docs/agents/isis-chroma-runpod-evidence.md § E.01.
What is shipped: the suite and the rubric, with their specs. What is owed: the runner, the score store, the dashboard scoring view, the external API reference, and every decision that would rest on a run. Nothing has been run and nothing has been scored. This document says which is which at every point.
1. Lanes, not workflows#
A lane is a question about quality, not a workflow. Several workflows may answer the same question, and the routing decision is taken per lane:
photoreal-portrait, nude-yoga, figure-study, dance-and-movement,
sets-and-environments, costume-and-wardrobe, anime-and-stylised,
typography.
nude-yoga and figure-study are the adult lanes — the ones that measure
the thing only the self-hosted lane may make.
2. The suite as shipped#
BENCHMARK_SUITE holds 14 entries across the eight lanes, 12 of them runnable
today:
| Lane | Entries | Workflows | Runnable |
|---|---|---|---|
| photoreal-portrait | 2 | chroma-txt2img, zimage-txt2img |
2 |
| nude-yoga | 2 | chroma-txt2img, zimage-txt2img |
2 |
| figure-study | 1 | chroma-txt2img |
1 |
| dance-and-movement | 3 | wan22-t2v, wan22-i2v, scail2-animate |
2 |
| sets-and-environments | 2 | qwen-txt2img, chroma-txt2img |
2 |
| costume-and-wardrobe | 1 | qwen-edit |
1 |
| anime-and-stylised | 2 | hidream-o1-txt2img ×2 |
1 |
| typography | 1 | qwen-txt2img |
1 |
Every entry carries a stable id, its lane, the workflowId it runs on, the
prompt and negative prompt, a fixed seed, the inputs it pins beyond prompt
and seed (the shape and anything that changes what is measured), its
contentRating, a consentId, an optional reference, whether it may also go
to the external API reference (apiReference), a status, and a one-line
measures saying what a scorer should look at.
Everything is a checked-in literal — prompt text, seed, shape, workflow id. Nothing is generated at run time, because a benchmark whose inputs drift is not a benchmark. Seeds are arbitrary but fixed, and deliberately different per entry so a lane's entries do not all collapse onto one sample of the noise.
2.1 Three rules the file is built around#
Adult prompts stay on adult_ok workflows and never leave RunPod. The three
adult-lane entries are apiReference: false and name adult_ok workflows, and
the spec asserts both. Policy is in the data, not in the runner's discipline: an
adult prompt cannot reach the external SFW reference through an edit to the
runner. Nine entries are cleared for that reference, all neutral.
References come from inside the suite, or the entry is blocked. Every
catalog workflow is prompt-driven, and operator self-consent
(consent_operator_isis_dev_studio, settled at C.12.14 as the correct record
for a typed prompt with no likeness) is the right consent for one. An entry
needing a start frame gets one produced by an earlier entry in this same
suite, so the material is Isis's own — wan22-i2v and qwen-edit both work
this way, and the spec enforces that a referenced entry is defined before the
entry that uses it.
An entry needing something Isis does not have is carried as blocked with the
tracker item that would unblock it, rather than dropped or pointed at stock
footage. Dropping it would make the lane read as covered; stock footage would
put uncleared likeness through the lane. Two entries are blocked:
dance-and-movement/scail2-animate-driven— needs a driving clip Oshun owns or has cleared (E.03.08).anime-and-stylised/netayume— holds the prompt and seed for a dedicated anime base (E.06.03).
A spec also refuses a lane whose only entries are blocked.
A renamed workflow or input fails the suite, not the run. The spec checks
every workflowId, every pinned input name and every enum value against the
real catalog JSON on disk.
2.2 What that spec caught#
On its first run the spec failed 12 of 93 assertions, all of them real defects in the suite's first draft:
- invented aspect values (
portrait,landscape,square) where the catalog namesportrait_896x1152,landscape_1152x896,square_1024; - a Z-Image entry using Chroma's aspect vocabulary;
- a Qwen typography entry pinned to a HiDream aspect (
portrait_3x4_1728x2304, which Qwen does not have); resolution: '480p'where Wan names480p_landscape_832x480.
Every one of those is either a submit refusal or a silently different shape — found for free instead of on a billed GPU minute.
3. The rubric as shipped#
The routing decision rests on these numbers, so the rubric has to be the kind of
thing two people would fill in the same way. BENCHMARK_RUBRIC_VERSION is
1.0.0, and every score records the version it was made under.
3.1 Six criteria, each anchored at both ends#
| Criterion | The question |
|---|---|
anatomy |
Is the body built correctly — count, placement and proportion of limbs, hands, joints? |
soft-tissue-motion |
Does flesh carry weight — does it rest, compress and settle the way a body does? |
prompt-adherence |
Did it make what the prompt asked for, including the parts that are easy to skip? |
identity-hold |
Is this recognisably the same person as the reference or first frame, all the way through? |
style-fidelity |
Does it hold the register the prompt asked for, consistently across the frame? |
set-and-wardrobe-consistency |
Did everything the prompt did not ask to change survive unchanged? |
The scale is 1–5, and every criterion says what a 1 and a 5 look like. A 1–5
with no anchors is a mood; with anchors, "3" means the same thing in March as it
did in January. For example, anatomy at 1 is "an extra or missing limb, fused
or six-fingered hands, a joint bending the wrong way"; at 5 it is "nothing a
life-drawing tutor would mark".
3.2 Criteria are per lane#
BENCHMARK_LANE_CRITERIA gives each lane only the criteria it can answer:
| Lane | Criteria |
|---|---|
| photoreal-portrait | anatomy, prompt-adherence, style-fidelity |
| nude-yoga | anatomy, soft-tissue-motion, prompt-adherence |
| figure-study | anatomy, soft-tissue-motion, prompt-adherence, style-fidelity |
| dance-and-movement | anatomy, soft-tissue-motion, prompt-adherence, identity-hold |
| sets-and-environments | prompt-adherence, set-and-wardrobe-consistency |
| costume-and-wardrobe | prompt-adherence, identity-hold, set-and-wardrobe-consistency |
| anime-and-stylised | anatomy, prompt-adherence, style-fidelity |
| typography | prompt-adherence |
Scoring soft-tissue motion on a typography poster produces a number that means nothing and then gets averaged, so a score for a criterion the lane does not use is rejected, not ignored.
3.3 Act-bleed is a defect count, not a quality#
ACT_BLEED_KINDS — sexual-pose, sexualised-framing,
unrequested-second-person — records material the prompt did not ask for.
Higher is worse, the opposite direction from every other criterion, so it is
typed differently rather than left as a trap for whoever averages the columns.
It is recorded for every lane, because "did it add something nobody asked
for" is always a fair question, and E.01.06 needs it reported per body-LoRA
rather than folded into an average.
An empty actBleed list means the rater looked and saw none — which is
different from not looking.
3.4 Blinding#
blindLabelFor(outputHash) gives the rater sample-<8 hex> derived from the
output bytes. It is stable for the same bytes and carries no workflow, model or
checkpoint. The rater does see the lane, because they need it to know which
criteria apply. Two runs of the same entry get different labels because their
bytes differ, which is what stops a rater recognising "the Chroma one" by
position.
3.5 A score is rejected, never coerced#
validateBenchmarkScore returns every problem rather than the first, and
refuses:
- no rater — "a score with no rater is not a human score";
- a
rubricVersionthat is not this build's; - an
outputHashthat is not a sha256; - a missing value for a criterion the lane is scored on;
- a value off the 1–5 scale;
- a criterion the lane does not use — "a number here would be averaged into a decision it cannot speak to";
- an act-bleed observation of an unknown kind.
A missing criterion silently stored as a default, or an out-of-lane criterion quietly dropped, would both put a number into the routing decision that nobody gave.
3.6 The aggregate reports what it does not know#
summariseBenchmarkLane computes per-criterion means over the outputs a
person actually scored, and reports scored and unscored beside every one
of them, so a summary of one score out of twelve cannot be read as a summary of
the lane. Where nobody has looked, the mean is null rather than 0 — a zero
would read as "scored, and terrible".
costPerAcceptedUsd divides everything the lane spent, rejects included, by
the number of outputs a person accepted: the rejected renders are part of the
price of the keeper. It is null when nothing has been accepted yet, because
dividing by zero and calling it expensive is not a measurement.
No automated judge stands in for a missing human score. An unscored output stays unscored.
4. What is owed#
Everything below is specified and not built. Nothing in this repository should be read as a benchmark result until these exist and a run has happened.
| Item | What is owed |
|---|---|
| E.01.02 | The score store in the generation API that persists BenchmarkScore records, and the dashboard view a person scores in. The rubric, the validation and the aggregate exist; nowhere to put a score does not. |
| E.01.03 | The runner: submits a suite run through the generation API as the operator tenant with consent ids attached, writes outputs under isis/benchmarks/<runId>/ in the Hetzner bucket, keeps them out of the Outputs gallery and every tenant-facing listing, and writes one results file per run (workflow, seed, measured cost, output hash, score state). Video jobs warm-chained. |
| E.01.04 | The external SFW API reference through @isis/seedance-provider: the suite's SFW prompts run once against the provider's current model id read at run time, with the per-call price taken from the response or invoice, never estimated. adult_ok prompts never go to an external API. Asks before spending. |
| E.01.05 | The run and the routing decision: a suite run per lane once that lane's E.02–E.06 items have rendered, each run's budget quoted and approved first. The owner then decides per lane — self-host, API, or both — recorded in a decisions block with the numbers behind it. |
| E.01.06 | The body-LoRA A/B: the nude-yoga, figure-study and dance prompts run with and without each body and motion option already in the library, at the same seeds, scored with act-bleed reported per option. The owner then decides keep or prune for each; a pruned option leaves the manifest, the library, the cache, every enum and the register in one commit. No sex-act LoRA is pulled for this. |
| E.01.07 | An own body-physics LoRA, scoped only. Its precondition is not agent-actionable: footage Oshun owns or has cleared, with every performer's consent in the registry, has to exist first. |
5. Running it, when the runner exists#
The shape the items fix, recorded here so a future run does not improvise:
- A lane is benchmarked only once its E.02–E.06 items have rendered — there is no point measuring a lane that is not finished.
- The run's budget is quoted and approved before it starts. A suite run is billed GPU time like any other work, and the hosted reference costs real money per call.
- Outputs go to
isis/benchmarks/<runId>/, never into the Outputs gallery or any tenant-facing listing. - A person scores blind against the rubric; unscored outputs stay unscored.
- The results, the scores and the cost per accepted output go into evidence § E.01.
- The owner decides per lane, and the decision is recorded with the numbers behind it — never inferred from the table.
6. Changing the suite or the rubric#
- Adding or editing an entry: it is a literal in
benchmark-suite.ts. The spec will fail on a workflow id, input name or enum value the catalog does not have, so run it before assuming an entry is valid. - An entry that needs material Isis does not own: mark it
blockedwith the tracker item that unblocks it. Do not drop it, and do not point it at uncleared material. - Changing a criterion, the scale or an anchor: bump
BENCHMARK_RUBRIC_VERSION. Existing scores then fail validation against the new build, which is the intended behaviour — a score made under different anchors is not comparable, and the aggregate must not silently mix them.