# The Isis benchmark suite

How Isis will decide, per lane, whether to keep generating on its own GPUs or
send that kind of work to an API — and what part of that machinery exists today.

The decision it serves is the owner's, taken on 14 September 2026:
**self-hosting versus APIs is decided per lane by measurement, not all or
nothing.** `adult_ok` lanes stay self-hosted by construction, because hosted
services refuse nudity.

Code: `libs/isis/workflows/src/benchmarks/benchmark-suite.ts` (the suite),
`benchmark-rubric.ts` (the rubric, the score record and the aggregate), and
their two spec files. Tracker: E.01.01–E.01.07. Evidence:
`docs/agents/isis-chroma-runpod-evidence.md` § E.01.

**What is shipped:** the suite and the rubric, with their specs. **What is
owed:** the runner, the score store, the dashboard scoring view, the external
API reference, and every decision that would rest on a run. Nothing has been run
and nothing has been scored. This document says which is which at every point.

## 1. Lanes, not workflows

A **lane** is a question about quality, not a workflow. Several workflows may
answer the same question, and the routing decision is taken per lane:

`photoreal-portrait`, `nude-yoga`, `figure-study`, `dance-and-movement`,
`sets-and-environments`, `costume-and-wardrobe`, `anime-and-stylised`,
`typography`.

`nude-yoga` and `figure-study` are the **adult lanes** — the ones that measure
the thing only the self-hosted lane may make.

## 2. The suite as shipped

`BENCHMARK_SUITE` holds **14 entries across the eight lanes, 12 of them runnable
today**:

| Lane                  | Entries | Workflows                                  | Runnable |
| --------------------- | ------- | ------------------------------------------ | -------- |
| photoreal-portrait    | 2       | `chroma-txt2img`, `zimage-txt2img`         | 2        |
| nude-yoga             | 2       | `chroma-txt2img`, `zimage-txt2img`         | 2        |
| figure-study          | 1       | `chroma-txt2img`                           | 1        |
| dance-and-movement    | 3       | `wan22-t2v`, `wan22-i2v`, `scail2-animate` | 2        |
| sets-and-environments | 2       | `qwen-txt2img`, `chroma-txt2img`           | 2        |
| costume-and-wardrobe  | 1       | `qwen-edit`                                | 1        |
| anime-and-stylised    | 2       | `hidream-o1-txt2img` ×2                    | 1        |
| typography            | 1       | `qwen-txt2img`                             | 1        |

Every entry carries a stable `id`, its lane, the `workflowId` it runs on, the
prompt and negative prompt, a fixed `seed`, the `inputs` it pins beyond prompt
and seed (the shape and anything that changes what is measured), its
`contentRating`, a `consentId`, an optional `reference`, whether it may also go
to the external API reference (`apiReference`), a `status`, and a one-line
`measures` saying what a scorer should look at.

**Everything is a checked-in literal** — prompt text, seed, shape, workflow id.
Nothing is generated at run time, because a benchmark whose inputs drift is not
a benchmark. Seeds are arbitrary but fixed, and deliberately different per entry
so a lane's entries do not all collapse onto one sample of the noise.

### 2.1 Three rules the file is built around

**Adult prompts stay on `adult_ok` workflows and never leave RunPod.** The three
adult-lane entries are `apiReference: false` and name `adult_ok` workflows, and
the spec asserts both. Policy is in the data, not in the runner's discipline: an
adult prompt cannot reach the external SFW reference through an edit to the
runner. Nine entries are cleared for that reference, all `neutral`.

**References come from inside the suite, or the entry is blocked.** Every
catalog workflow is prompt-driven, and operator self-consent
(`consent_operator_isis_dev_studio`, settled at C.12.14 as the correct record
for a typed prompt with no likeness) is the right consent for one. An entry
needing a start frame gets one produced by an **earlier entry in this same
suite**, so the material is Isis's own — `wan22-i2v` and `qwen-edit` both work
this way, and the spec enforces that a referenced entry is defined before the
entry that uses it.

An entry needing something Isis does not have is carried as `blocked` with the
tracker item that would unblock it, rather than dropped or pointed at stock
footage. Dropping it would make the lane read as covered; stock footage would
put uncleared likeness through the lane. Two entries are blocked:

- `dance-and-movement/scail2-animate-driven` — needs a driving clip Oshun owns
  or has cleared (**E.03.08**).
- `anime-and-stylised/netayume` — holds the prompt and seed for a dedicated
  anime base (**E.06.03**).

A spec also refuses a lane whose _only_ entries are blocked.

**A renamed workflow or input fails the suite, not the run.** The spec checks
every `workflowId`, every pinned input name and every enum value against the
real catalog JSON on disk.

### 2.2 What that spec caught

On its first run the spec failed **12 of 93 assertions**, all of them real
defects in the suite's first draft:

- invented aspect values (`portrait`, `landscape`, `square`) where the catalog
  names `portrait_896x1152`, `landscape_1152x896`, `square_1024`;
- a Z-Image entry using Chroma's aspect vocabulary;
- a Qwen typography entry pinned to a HiDream aspect (`portrait_3x4_1728x2304`,
  which Qwen does not have);
- `resolution: '480p'` where Wan names `480p_landscape_832x480`.

Every one of those is either a submit refusal or a silently different shape —
found for free instead of on a billed GPU minute.

## 3. The rubric as shipped

The routing decision rests on these numbers, so the rubric has to be the kind of
thing two people would fill in the same way. `BENCHMARK_RUBRIC_VERSION` is
`1.0.0`, and every score records the version it was made under.

### 3.1 Six criteria, each anchored at both ends

| Criterion                      | The question                                                                               |
| ------------------------------ | ------------------------------------------------------------------------------------------ |
| `anatomy`                      | Is the body built correctly — count, placement and proportion of limbs, hands, joints?     |
| `soft-tissue-motion`           | Does flesh carry weight — does it rest, compress and settle the way a body does?           |
| `prompt-adherence`             | Did it make what the prompt asked for, including the parts that are easy to skip?          |
| `identity-hold`                | Is this recognisably the same person as the reference or first frame, all the way through? |
| `style-fidelity`               | Does it hold the register the prompt asked for, consistently across the frame?             |
| `set-and-wardrobe-consistency` | Did everything the prompt did **not** ask to change survive unchanged?                     |

The scale is 1–5, and **every criterion says what a 1 and a 5 look like**. A 1–5
with no anchors is a mood; with anchors, "3" means the same thing in March as it
did in January. For example, `anatomy` at 1 is "an extra or missing limb, fused
or six-fingered hands, a joint bending the wrong way"; at 5 it is "nothing a
life-drawing tutor would mark".

### 3.2 Criteria are per lane

`BENCHMARK_LANE_CRITERIA` gives each lane only the criteria it can answer:

| Lane                  | Criteria                                                      |
| --------------------- | ------------------------------------------------------------- |
| photoreal-portrait    | anatomy, prompt-adherence, style-fidelity                     |
| nude-yoga             | anatomy, soft-tissue-motion, prompt-adherence                 |
| figure-study          | anatomy, soft-tissue-motion, prompt-adherence, style-fidelity |
| dance-and-movement    | anatomy, soft-tissue-motion, prompt-adherence, identity-hold  |
| sets-and-environments | prompt-adherence, set-and-wardrobe-consistency                |
| costume-and-wardrobe  | prompt-adherence, identity-hold, set-and-wardrobe-consistency |
| anime-and-stylised    | anatomy, prompt-adherence, style-fidelity                     |
| typography            | prompt-adherence                                              |

Scoring soft-tissue motion on a typography poster produces a number that means
nothing and then gets averaged, so a score for a criterion the lane does not use
is **rejected, not ignored**.

### 3.3 Act-bleed is a defect count, not a quality

`ACT_BLEED_KINDS` — `sexual-pose`, `sexualised-framing`,
`unrequested-second-person` — records material the prompt did not ask for.
Higher is worse, the opposite direction from every other criterion, so it is
typed differently rather than left as a trap for whoever averages the columns.
It is recorded for **every** lane, because "did it add something nobody asked
for" is always a fair question, and E.01.06 needs it reported per body-LoRA
rather than folded into an average.

An empty `actBleed` list means the rater looked and saw none — which is
different from not looking.

### 3.4 Blinding

`blindLabelFor(outputHash)` gives the rater `sample-<8 hex>` derived from the
output bytes. It is stable for the same bytes and carries no workflow, model or
checkpoint. The rater **does** see the lane, because they need it to know which
criteria apply. Two runs of the same entry get different labels because their
bytes differ, which is what stops a rater recognising "the Chroma one" by
position.

### 3.5 A score is rejected, never coerced

`validateBenchmarkScore` returns **every** problem rather than the first, and
refuses:

- no rater — "a score with no rater is not a human score";
- a `rubricVersion` that is not this build's;
- an `outputHash` that is not a sha256;
- a missing value for a criterion the lane is scored on;
- a value off the 1–5 scale;
- a criterion the lane does not use — "a number here would be averaged into a
  decision it cannot speak to";
- an act-bleed observation of an unknown kind.

A missing criterion silently stored as a default, or an out-of-lane criterion
quietly dropped, would both put a number into the routing decision that nobody
gave.

### 3.6 The aggregate reports what it does not know

`summariseBenchmarkLane` computes per-criterion means **over the outputs a
person actually scored**, and reports `scored` and `unscored` beside every one
of them, so a summary of one score out of twelve cannot be read as a summary of
the lane. Where nobody has looked, the mean is `null` rather than `0` — a zero
would read as "scored, and terrible".

`costPerAcceptedUsd` divides **everything the lane spent, rejects included**, by
the number of outputs a person accepted: the rejected renders are part of the
price of the keeper. It is `null` when nothing has been accepted yet, because
dividing by zero and calling it expensive is not a measurement.

**No automated judge stands in for a missing human score.** An unscored output
stays unscored.

## 4. What is owed

Everything below is specified and not built. Nothing in this repository should
be read as a benchmark result until these exist and a run has happened.

| Item        | What is owed                                                                                                                                                                                                                                                                                                                                                                                           |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **E.01.02** | The **score store** in the generation API that persists `BenchmarkScore` records, and the **dashboard view** a person scores in. The rubric, the validation and the aggregate exist; nowhere to put a score does not.                                                                                                                                                                                  |
| **E.01.03** | The **runner**: submits a suite run through the generation API as the operator tenant with consent ids attached, writes outputs under `isis/benchmarks/<runId>/` in the Hetzner bucket, keeps them out of the Outputs gallery and every tenant-facing listing, and writes one results file per run (workflow, seed, measured cost, output hash, score state). Video jobs warm-chained.                 |
| **E.01.04** | The **external SFW API reference** through `@isis/seedance-provider`: the suite's SFW prompts run once against the provider's current model id read at run time, with the per-call price taken from the response or invoice, **never estimated**. `adult_ok` prompts never go to an external API. Asks before spending.                                                                                |
| **E.01.05** | The **run and the routing decision**: a suite run per lane once that lane's E.02–E.06 items have rendered, each run's budget quoted and approved first. The owner then decides per lane — self-host, API, or both — recorded in a decisions block with the numbers behind it.                                                                                                                          |
| **E.01.06** | The **body-LoRA A/B**: the nude-yoga, figure-study and dance prompts run with and without each body and motion option already in the library, at the same seeds, scored with act-bleed reported per option. The owner then decides keep or prune for each; a pruned option leaves the manifest, the library, the cache, every enum and the register in one commit. No sex-act LoRA is pulled for this. |
| **E.01.07** | An **own body-physics LoRA, scoped only**. Its precondition is not agent-actionable: footage Oshun owns or has cleared, with every performer's consent in the registry, has to exist first.                                                                                                                                                                                                            |

## 5. Running it, when the runner exists

The shape the items fix, recorded here so a future run does not improvise:

1. A lane is benchmarked only once its E.02–E.06 items have rendered — there is
   no point measuring a lane that is not finished.
2. The run's budget is quoted and approved **before** it starts. A suite run is
   billed GPU time like any other work, and the hosted reference costs real
   money per call.
3. Outputs go to `isis/benchmarks/<runId>/`, never into the Outputs gallery or
   any tenant-facing listing.
4. A person scores blind against the rubric; unscored outputs stay unscored.
5. The results, the scores and the cost per accepted output go into evidence §
   E.01.
6. The owner decides per lane, and the decision is recorded with the numbers
   behind it — never inferred from the table.

## 6. Changing the suite or the rubric

- **Adding or editing an entry**: it is a literal in `benchmark-suite.ts`. The
  spec will fail on a workflow id, input name or enum value the catalog does not
  have, so run it before assuming an entry is valid.
- **An entry that needs material Isis does not own**: mark it `blocked` with the
  tracker item that unblocks it. Do not drop it, and do not point it at
  uncleared material.
- **Changing a criterion, the scale or an anchor**: bump
  `BENCHMARK_RUBRIC_VERSION`. Existing scores then fail validation against the
  new build, which is the intended behaviour — a score made under different
  anchors is not comparable, and the aggregate must not silently mix them.
