# Evaluator operations Phase 182, `182.C.39.05`. Acquisition, calibration, thresholds, failure interpretation, fairness slices, upgrades, and rollback. Which evaluators and where they run are decided elsewhere and not repeated here: see `sync-evaluator-selection.md` and `evaluator-hosting-decision.md`. ## 1. Acquisition **An evaluator is a pinned artifact, not a library name.** Weights, digest, version, and the date they were fetched (`human-video-model-acquisition.ts`, `human-video-model-record.ts`). A dependency that silently updates its weights invalidates **every threshold ever fitted against it**, quietly, and the first symptom is a pass rate that moved for no reason anybody can name. A model that cannot be pinned cannot be used for a release decision. It can be used for exploration, and the distinction has to survive contact with a deadline. ## 2. Calibration Calibration fits a threshold to a corpus, which means **the threshold inherits the corpus's biases** — if a corpus is mostly well-lit frontal faces, the resulting cutoff is a cutoff for well-lit frontal faces wearing a general name. That is what the fairness slices exist to find, and why calibration and slicing are one procedure rather than two. Split by subject and scenario, never by clip (`human-video-corpus-splits.ts`). A random clip split leaks the subject into both halves, and the number that comes back is "can it recognise this person, whom it has already seen". ## 3. Thresholds: what is measured and what is provisional Two kinds live in this codebase and they are not interchangeable. **Measured.** Derived from a committed measurement, with the evidence path in the code — for example the identity limits from `182.C.19.08`: 21 px of motion blur leaves 4% of identity margin, and at 31 px a face scored 0.251 against its own clean self, under the 0.282 two-different-people baseline. **Provisional.** Chosen to catch a degenerate case, not fitted to rated material, and exported under a name that says so. There are **nineteen provisional threshold exports** today: | Module | Export | | -------------------------------------- | --------------------------------------------------------------- | | `human-video-appearance.ts` | `PROVISIONAL_APPEARANCE_THRESHOLDS` | | `human-video-active-scene-cases.ts` | `PROVISIONAL_MIN_CONCURRENCY` | | `human-video-audio-defects.ts` | `PROVISIONAL_DEFECT_THRESHOLDS` | | `human-video-clip-reconciliation.ts` | `PROVISIONAL_RECONCILIATION_THRESHOLDS` | | `human-video-conversation-dynamics.ts` | `PROVISIONAL_MIN_GAP_SPREAD_MS` | | `human-video-desync-patterns.ts` | `PROVISIONAL_THRESHOLDS` | | `human-video-face-swap.ts` | `PROVISIONAL_SWAP_THRESHOLDS` | | `human-video-frame-integrity.ts` | `PROVISIONAL_TEAR_THRESHOLDS` | | `human-video-loudness.ts` | `PROVISIONAL_MIN_ARRIVAL_CREST_DB` | | `human-video-measurability.ts` | `PROVISIONAL_MIN_MEASURABLE_FRACTION` | | `human-video-motion-quality.ts` | `PROVISIONAL_MOTION_THRESHOLDS` | | `human-video-offscreen-dialogue.ts` | `PROVISIONAL_YAW_BOUNDARIES`, `PROVISIONAL_MIN_SCORED_FRACTION` | | `human-video-pronunciation.ts` | `PROVISIONAL_GOP_FLOOR` | | `human-video-scene-rejection.ts` | `PROVISIONAL_REJECTION_THRESHOLDS` | | `human-video-speaker-attribution.ts` | `PROVISIONAL_ATTRIBUTION_THRESHOLDS` | | `human-video-speaker-binding.ts` | `PROVISIONAL_FALSE_MOTION_RATIO` | | `human-video-visual-integrity.ts` | `PROVISIONAL_INTEGRITY_THRESHOLDS` | | `human-video-word-alignment.ts` | `PROVISIONAL_TIMING_THRESHOLDS` | **Every release decision currently rests on that list.** `182.C.17.08` is the item that replaces them with rated numbers, and it is blocked on the corpus in `182.C.33`. Nobody should quote any of these as a measured limit. ## 4. Failure interpretation **A low score and an unmeasurable window are different outcomes**, and merging them is the most expensive mistake available here: a shot of the back of somebody's head produces no findings, and "no findings" walks into a release gate looking exactly like "no problems" (`human-video-measurability.ts`). "Out of sync" is not one finding either. Six patterns, because they need six different repairs (`human-video-desync-patterns.ts`): | Pattern | What it is | What fixes it | | --------------------- | ---------------------------------------- | -------------------------- | | `global-offset` | the whole clip is out by the same amount | a mux fix | | `local-drift` | the error grows through the clip | a rate conform | | `intermittent-desync` | fine except for a bad stretch | retry that shot | | `frozen-mouth` | no mouth motion under speech | a different model | | `motion-in-silence` | mouth moving with nothing to say | a different model | | `edit-boundary` | the error appears at a cut | the assembly, not the take | A single score cannot distinguish these, which is why the per-window series is kept rather than averaged away. **A median offset is only a global offset when the spread is small**; an average of scattered offsets is a number that describes nothing. ## 5. Fairness slices Eight slices are declared: dark skin, varied lighting, facial hair, head coverings, makeup, glasses, age, and disability-related differences (`human-video-fairness-slices.ts`). **Six of the eight are declared rather than measured by design**, and no bucket is powered today — the minimum is 30 subjects per bucket and the corpus has none, so the module refuses to report a rate from any of them. **Skin tone cannot be a measured cohort, and this repo measured why**: a one-stop exposure change moved a single subject 58.2° of ITA, against a 63.7° gap between two different people. A skin-tone cohort computed from pixels is therefore mostly a **lighting** cohort, and a fairness dashboard built on it would attribute a lighting failure to skin tone. Tone is declared by the subject, never inferred (`human-video-cohorts.ts`). ## 6. Upgrades **An evaluator upgrade invalidates the evidence gathered under the old one.** The new version scores differently by definition — that is why it is an upgrade — so every threshold fitted against the previous version is unfitted, and every cohort claim substantiated with it is unsubstantiated. The procedure is therefore a re-benchmark, not a swap: 1. Pin and record the new artifact (§1). 2. Re-run the calibration split (§2); do not reuse the old thresholds. 3. Re-establish the cohort claims that gated release — `182.C.34.07`'s 299 clean renders per cohort, priced at $598 per cohort at eight-second takes. 4. Only then move traffic. Running the two in shadow first costs one extra evaluation per take and is the only way to learn how far apart they score before the answer matters. ## 7. Rollback Rolling an evaluator back has the same shape as rolling code back (`182.C.38.06`): **move the evaluator, never the record**. - Every measurement carries the evaluator version that produced it. A rollback does not re-grade old measurements under the older evaluator — that would produce verdicts no version ever issued. - A candidate measured under the newer evaluator stays measured under it. If its release decision is no longer trusted, it is **re-measured** and gets a new record, not an edited one. - The release proof names the evaluator version, so a proof written under the version you just rolled back is still readable — and still true about what was measured at the time. ## Where this is still theory No evaluator has scored a real generated clip in this pipeline. The thresholds above are provisional for exactly that reason, and the fairness buckets are empty. `182.C.17.08` and `182.C.33` are the two items that change this page.