Last updated: 2026-09-10
The machine-readable form of this record is
apps/oshun/bff/src/generation/human-video-sync-evaluators.ts; the digests,
licences and throughput numbers there are the ones the acquisition step in
182.C.17.02 fetches against, and the spec beside it verifies them against the
weights actually staged on this box. This document is the reasoning.
The selection rule#
Two evaluators, chosen to fail differently. Two measures that share a failure mode are one measure with extra cost, and the entire reason for having two is that a release claim should not rest on a single model's blind spot.
| Evaluator | Family | Strong where | Blind where |
|---|---|---|---|
syncnet-v2 |
learned embedding | Poor lighting, low resolution, unusual faces — it never has to locate a landmark | Profile views and small faces, and it does not lower its own confidence when it degrades |
lip-envelope-correlation |
geometric | Frozen mouths, mouth motion during silence, and it says why | Small, turned or occluded faces — but it reports landmark confidence, so this is visible |
That mirroring is the point. Where one is blind the other sees, and a window
neither can measure is coverage (182.C.17.06), reported as such rather
than scored as a pass.
What was selected#
1. SyncNet v2 — the LSE-C / LSE-D baseline#
- Source:
joonson/syncnet_python - Licence: MIT, copyright Joon Son Chung. The licence text is on disk at
~/.cache/oshun-human-video/LICENSE.mdand this record was written from it, not from a repository badge. - Weights:
syncnet_v2.model, 54,573,114 bytes, sha256961e8696f888fce4f3f3a6c3d5b3267cf5b343100b238e79b2659bff2c605442, fromhttps://www.robots.ox.ac.uk/~vgg/software/lipsync/data/syncnet_v2.model. Loads withstrict=True: 13.63 M parameters. - Measured throughput (2026-09-10, this box, 4 threads,
torch.no_grad()): lip stream 0.80 s for a batch of 20[3, 5, 224, 224]windows; audio stream 0.31 s for 20[1, 13, 20]. About 1.4 CPU-seconds per second of 25 fps video for both streams. Evidence:evidence/2026-09-10-cpu-sync-and-alignment.
2. Lip-envelope correlation — the geometric complement#
Mouth aperture from MediaPipe face landmarks against the speech envelope, cross-correlated over candidate lags.
- Licence: Apache-2.0 (the method is arithmetic; the licence that matters is the landmark model's).
- Weights:
face_landmarker.task, 3,758,596 bytes, sha25664184e229b263107bc2b804c6625db1341ff2bb731874b0bcc2fe6544e0bc9ff.
It is a method rather than a second learned model, and that is a deliberate choice rather than a shortfall. It is fully interpretable, carries no licence encumbrance, runs in CI on any host, and — the decisive property — it detects directly the two failures an embedding scores only indirectly: a mouth that never moves while words play, and a mouth moving through silence. A second opaque learned model that cannot be run here would measure less and prove less.
Why not a second learned model, yet#
Recorded so the question is answered from this record rather than re-opened from scratch. None of these has been benchmarked here and nothing below claims it has; the blockers are licensing and hardware.
| Candidate | What stands in the way |
|---|---|
| AV-HuBERT | CC-BY-NC — non-commercial. Disqualified for a product creators pay for, whatever its accuracy. A licensing fact, not a performance judgement. |
| Wav2Lip expert discriminator | Same SyncNet lineage as the baseline, so its failures would not be independent. A second evaluator that fails where the first fails is one evaluator. |
| SparseSync | Licence and CPU throughput both unverified here. Adopting it means measuring it on this box first, the way SyncNet v2 was, and recording what was seen. |
Independent agreement#
A release claim needs two evaluators that both actually measured, and
assertIndependentAgreement enforces it. Two rules carry the weight:
- An evaluator that could not measure a window does not count toward the two. An unmeasured window is coverage, not agreement.
- Evaluators from a shared lineage do not count as two. Their agreement is one measurement reported twice, and the disagreement that would have caught the failure never happens.
The human calibration protocol#
A score is a number until somebody says which numbers a viewer notices. That is
what this is for, and it is executable — assertCalibrationDesign refuses a
design that cannot support the threshold it would set, so "we calibrated against
human ratings" cannot come to mean nine ratings from one person.
Blinding. Raters see the clip and nothing else — not the evaluator's score, not the model that produced it. Checked first, because an unblinded rating is not a weaker measurement of perception; it is a measurement of the number the rater was shown.
Stimulus strata. Every dimension along which sync perception or sync measurement is known to move, each with its own cell: language, skin tone, facial hair, head pose, speaking style, subject motion, compression level, shot size. A stratum nobody sampled is refused rather than assumed to behave like the others — a threshold that was never tested against it is a threshold that does not hold across it.
Power. At least 30 clips per cell, because that is roughly where a proportion's confidence interval narrows enough to separate a 90% pass rate from an 80% one, which is the size of difference a threshold decision turns on. At least 3 raters per clip, because below three there is no agreement statistic — two raters produce a number that cannot distinguish "they agree" from "they both guessed". The protocol uses 5.
Agreement. Krippendorff's alpha at or above 0.667, the conventional floor for tentative conclusions. Below it the raters are not measuring the same thing, and a threshold fitted to them encodes their disagreement rather than the perception.
Scale. Forced choice per clip: the speech matches the mouth / the speech is ahead / the speech is behind / cannot tell. The last one is a real answer and is recorded as coverage, not as a pass — which is the same rule the evaluators follow, so the calibration and the thing being calibrated count the same events the same way.
The thresholds themselves are set by 182.C.17.08, executing this design. They
are not set here, and no number in this document is a threshold.