# Sync evaluator selection, licensing, and human calibration (`182.C.17.01`)

Last updated: 2026-09-10

The machine-readable form of this record is
`apps/oshun/bff/src/generation/human-video-sync-evaluators.ts`; the digests,
licences and throughput numbers there are the ones the acquisition step in
`182.C.17.02` fetches against, and the spec beside it verifies them against the
weights actually staged on this box. This document is the reasoning.

## The selection rule

Two evaluators, chosen to **fail differently**. Two measures that share a
failure mode are one measure with extra cost, and the entire reason for having
two is that a release claim should not rest on a single model's blind spot.

| Evaluator                  | Family            | Strong where                                                                     | Blind where                                                                              |
| -------------------------- | ----------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `syncnet-v2`               | learned embedding | Poor lighting, low resolution, unusual faces — it never has to locate a landmark | Profile views and small faces, and it does not lower its own confidence when it degrades |
| `lip-envelope-correlation` | geometric         | Frozen mouths, mouth motion during silence, and it says _why_                    | Small, turned or occluded faces — but it reports landmark confidence, so this is visible |

That mirroring is the point. Where one is blind the other sees, and a window
neither can measure is **coverage** (`182.C.17.06`), reported as such rather
than scored as a pass.

## What was selected

### 1. SyncNet v2 — the LSE-C / LSE-D baseline

- **Source**: `joonson/syncnet_python`
- **Licence**: MIT, copyright Joon Son Chung. The licence text is on disk at
  `~/.cache/oshun-human-video/LICENSE.md` and this record was written from it,
  not from a repository badge.
- **Weights**: `syncnet_v2.model`, 54,573,114 bytes, sha256
  `961e8696f888fce4f3f3a6c3d5b3267cf5b343100b238e79b2659bff2c605442`, from
  `https://www.robots.ox.ac.uk/~vgg/software/lipsync/data/syncnet_v2.model`.
  Loads with `strict=True`: 13.63 M parameters.
- **Measured throughput** (2026-09-10, this box, 4 threads, `torch.no_grad()`):
  lip stream 0.80 s for a batch of 20 `[3, 5, 224, 224]` windows; audio stream
  0.31 s for 20 `[1, 13, 20]`. About **1.4 CPU-seconds per second of 25 fps
  video** for both streams. Evidence:
  `evidence/2026-09-10-cpu-sync-and-alignment`.

### 2. Lip-envelope correlation — the geometric complement

Mouth aperture from MediaPipe face landmarks against the speech envelope,
cross-correlated over candidate lags.

- **Licence**: Apache-2.0 (the method is arithmetic; the licence that matters is
  the landmark model's).
- **Weights**: `face_landmarker.task`, 3,758,596 bytes, sha256
  `64184e229b263107bc2b804c6625db1341ff2bb731874b0bcc2fe6544e0bc9ff`.

It is a method rather than a second learned model, and that is a deliberate
choice rather than a shortfall. It is fully interpretable, carries no licence
encumbrance, runs in CI on any host, and — the decisive property — it detects
directly the two failures an embedding scores only indirectly: a mouth that
never moves while words play, and a mouth moving through silence. A second
opaque learned model that cannot be run here would measure less and prove less.

## Why not a second learned model, yet

Recorded so the question is answered from this record rather than re-opened from
scratch. None of these has been benchmarked here and nothing below claims it
has; the blockers are licensing and hardware.

| Candidate                        | What stands in the way                                                                                                                                  |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **AV-HuBERT**                    | CC-BY-NC — non-commercial. Disqualified for a product creators pay for, whatever its accuracy. A licensing fact, not a performance judgement.           |
| **Wav2Lip expert discriminator** | Same SyncNet lineage as the baseline, so its failures would not be independent. A second evaluator that fails where the first fails is one evaluator.   |
| **SparseSync**                   | Licence and CPU throughput both unverified here. Adopting it means measuring it on this box first, the way SyncNet v2 was, and recording what was seen. |

## Independent agreement

A release claim needs **two evaluators that both actually measured**, and
`assertIndependentAgreement` enforces it. Two rules carry the weight:

1. An evaluator that could not measure a window does **not** count toward the
   two. An unmeasured window is coverage, not agreement.
2. Evaluators from a shared lineage do not count as two. Their agreement is one
   measurement reported twice, and the disagreement that would have caught the
   failure never happens.

## The human calibration protocol

A score is a number until somebody says which numbers a viewer notices. That is
what this is for, and it is executable — `assertCalibrationDesign` refuses a
design that cannot support the threshold it would set, so "we calibrated against
human ratings" cannot come to mean nine ratings from one person.

**Blinding.** Raters see the clip and nothing else — not the evaluator's score,
not the model that produced it. Checked first, because an unblinded rating is
not a weaker measurement of perception; it is a measurement of the number the
rater was shown.

**Stimulus strata.** Every dimension along which sync perception or sync
measurement is known to move, each with its own cell: language, skin tone,
facial hair, head pose, speaking style, subject motion, compression level, shot
size. A stratum nobody sampled is refused rather than assumed to behave like the
others — a threshold that was never tested against it is a threshold that does
not hold across it.

**Power.** At least 30 clips per cell, because that is roughly where a
proportion's confidence interval narrows enough to separate a 90% pass rate from
an 80% one, which is the size of difference a threshold decision turns on. At
least 3 raters per clip, because below three there is no agreement statistic —
two raters produce a number that cannot distinguish "they agree" from "they both
guessed". The protocol uses 5.

**Agreement.** Krippendorff's alpha at or above 0.667, the conventional floor
for tentative conclusions. Below it the raters are not measuring the same thing,
and a threshold fitted to them encodes their disagreement rather than the
perception.

**Scale.** Forced choice per clip: _the speech matches the mouth_ / _the speech
is ahead_ / _the speech is behind_ / _cannot tell_. The last one is a real
answer and is recorded as coverage, not as a pass — which is the same rule the
evaluators follow, so the calibration and the thing being calibrated count the
same events the same way.

The thresholds themselves are set by `182.C.17.08`, executing this design. They
are not set here, and no number in this document is a threshold.
