Evidence for 182.C.17 (sync evaluators), 182.C.18.02 (word alignment), and
the phase-level hardware constraint. Third correction of the day, same
cause: all thirteen 182.C.17 rows, all eight 182.C.20 rows, and all eight
182.C.32 rows carried one copied blocker, "No GPU host; needs a RunPod worker
per 182.C.17.13", and 182.C.18.02 was blocked on "a forced aligner on the
hosted GPU worker". None of those items had been tried. Both premises were
tested here and both are false for the evaluator side.
Machine#
Linux, 15 GiB RAM, 4 cores, no GPU (nvidia-smi absent; torch 2.14.0+cpu,
faster-whisper 1.2.1, transformers 5.17.0, onnxruntime 1.25.1 installed
system-wide). free -m before the runs: 10.9 GB available.
1. SyncNet v2 runs on this CPU#
joonson/syncnet_python — the LSE-C / LSE-D baseline the phase names in
182.C.17.01. MIT licence (LICENSE.md, copyright Joon Son Chung). Weights
syncnet_v2.model, 54,573,114 bytes, sha256
961e8696f888fce4f3f3a6c3d5b3267cf5b343100b238e79b2659bff2c605442, fetched from
https://www.robots.ox.ac.uk/~vgg/software/lipsync/data/syncnet_v2.model. Loads
with strict=True: 13.63 M parameters.
Forward pass on CPU, 4 threads, torch.no_grad(), batch of 20 windows:
| Stream | Input shape | Output | Time |
|---|---|---|---|
lip (forward_lip) |
[20, 3, 5, 224, 224] |
[20, 1024] |
0.80 s |
audio (forward_aud) |
[20, 1, 13, 20] |
[20, 1024] |
0.31 s |
| per second of 25 fps video (25 five-frame windows) | — | — | ≈ 1.0 s CPU (lip) + 0.4 s (audio) |
A 10 s candidate therefore costs about 15 s of CPU to score end to end, before
face tracking. That is a throughput number for 182.C.17.12, not a blocker. The
hosted GPU worker in 182.C.17.13 is a production fan-out decision to be taken
from measured throughput, not a prerequisite for building the evaluator.
Not tested here: a real talking-face clip through the full pipeline (face crop → MFCC → SyncNet → offset/confidence). No generated human video exists in the repo yet, and producing one is provider spend under prohibition 2 (consent binding before any reference upload). The forward pass on correctly shaped tensors proves the compute premise only.
2. A CTC forced aligner runs on this CPU, and it is accurate#
facebook/wav2vec2-base-960h through transformers (Apache-2.0; the
masked_spec_embed weight is reported missing at load, which only affects
training-time masking), with the standard CTC trellis and backtracking. 20.1 ms
frames.
Ground truth#
Timestamps from a chat model are generated, and Whisper's are inferred from
cross-attention; to measure either you need onsets known to the sample. So:
Kokoro-82M (hexgrad/Kokoro-82M, cached locally, voice af_heart) synthesized
six words separately, each trimmed to its first and last sample above
|0.01|, then concatenated with exactly 600 ms of digital silence between them.
Every onset is therefore known exactly (synth-truth.json;
known-onset-clip-16k.wav, 7.23 s).
Results (align-accuracy.json)#
| Aligner | CPU time | Onset error per word (ms) | Max abs onset | Word-end error (ms) |
|---|---|---|---|---|
| wav2vec2 CTC forced alignment | 1.29 s for 7.23 s | +81, +65, +17, +17, +62, +35 | 81 ms | −96 … −141 (consistently early) |
faster-whisper tiny.en, word_timestamps |
2.0 s (3.95 s clip) | −600, +102, +532, +151, +113 | 600 ms | merged "forty dollars" → "$40" |
faster-whisper base.en |
5.7 s | −600, −58, −128, −49, −87, −116 | 600 ms | −218 … −352 |
faster-whisper small.en |
15.3 s | −600, +182, +52, +171, +33 | 600 ms | merged "forty dollars" → "$40" |
Against the thresholds in 182.C.03.04 (ATSC IS-191 target +15/−45 ms; ITU-R
BT.1359-1 reject +45/−125 ms):
- CTC is inside the ITU reject boundary on every word, raw. Its error is a
systematic late bias (every onset positive, 17–81 ms), so a calibrated
constant corrects most of it. That calibration was run — it is the first step
of
182.C.18.02— and it corrects an earlier claim on this page, which said the constant brings CTC inside the ATSC window. It does not.aligner-calibration.json(tools/human-video/evaluator/calibrate_aligner.py, reproducing the onsets above exactly): median bias +48.288 ms, so the correction is −48.288 ms; the residual scatter that remains is ±32.87 ms (worst lead −31.68, worst lag +32.87). ATSC IS-191 allows only 15 ms of lead, and the corrected worst lead is 31.68 — outside ATSC. It is comfortably inside ITU-R BT.1359-1 (+45/−125) with ~30% margin on the tighter side. That ±33 ms is the aligner's own noise floor and therefore the floor on any timing claim built from it. Word ends are consistently ~100–140 ms early (CTC emits a token before the sound finishes), so use onsets, not ends, for offset work. - Whisper's timestamps are disqualified, like the OpenRouter chat models' in
../2026-09-10-openrouter-asr/: the first word is snapped to 0.0 (600 ms early, every model size), mid-clip onsets are off by up to 532 ms, and two of three sizes merged two spoken words into one orthographic token, which also breaks word counting. Whisper remains fine for transcription; it is not an aligner. - On the real
probe.wav(24 kHz TTS,../2026-09-10-openrouter-asr/), CTC aligned the spoken wordsforty two dollarsas three words in 0.68 s (fw-words.jsonhas the Whisper timings for the same clip); a chat model returns$42and the numbers-to-words expansion indialogue.tsmust run before alignment.
Pronunciation, from the same posteriors (pronunciation-gop.json)#
182.C.18.03 asks a different question from 182.C.18.01: not whether the
recognizer wrote the right letters, but whether the ACOUSTICS support them. A
recognizer's language model repairs a mangled brand name into the spelling it
expects, so a clean transcript is not evidence the name was said right.
The measure is Witt & Young goodness of pronunciation over the same CTC model,
GOP = log P(canonical | emission frame) - max_s log P(s | emission frame). At
most 0; 0 means the acoustic model would itself have chosen that character.
| Clip | Correct words | One word substituted, same audio |
|---|---|---|
known-onset-clip-16k.wav |
all six at exactly 0.0 | tuesday→thursday −15.32, costs→coasts −9.62, forty→fourteen −11.17, engine→enjoying −20.18 |
../2026-09-10-openrouter-asr/probe.wav (24k→16k) |
all ten at exactly 0.0 | — |
In every substitution the other five words stayed at exactly 0.0, so the fault
localises to a word and to a character within it (thursday fails at R, where
the model preferred E).
The finding, and it constrains how this can be used. The separation is
total, so any floor in the gap separates these cases and the measurement does
not say which value is right. More importantly the measure SATURATES: two
different voices and two different sentences all sit at exactly 0.0, so there is
no headroom above zero in which to rank a perfect pronunciation against an
accented but acceptable one. It is a detector, not a grade. Both fixtures are
clean synthetic English, so they do not establish a threshold for real human
speech at all — that needs the 182.C.33 rated corpus, the same blocker as
182.C.17.08. The shipped floor is −3.0 and is labelled provisional in code,
placed nearer the correct end deliberately, because a false "mispronounced" on a
real accent rejects a take a human would accept.
Locale is the other half. wav2vec2-base-960h is LibriSpeech English. Given
a Yoruba name it has no symbols for those sounds, and a low GOP then means the
model does not know the language rather than that the speaker said it wrongly.
Those terms are reported UNMEASURABLE with no number attached and routed to
182.C.18.07; unmeasurable and failed must never look alike.
A defect this work found in the aligner committed hours earlier. tokenize
silently dropped any character outside the model vocabulary. Aligning
"costs 40 dollars" against audio saying "costs forty dollars" dropped both
digits, put 40 463 ms late, and lost dollars from the output entirely. It
now refuses with unalignable_character, and merge_words groups by the labels
the tokenizer produced instead of counting transcript characters in parallel —
the two walks agreeing was the assumption that broke. The 182.C.18.02
calibration onsets are byte-identical after the rewrite.
Per-item status this evidence supports#
| Item | Was | Is |
|---|---|---|
182.C.17.01–.12 |
blocked-local, "No GPU host" |
not-started; SyncNet v2 measured on this CPU; second evaluator still to be selected and licensed |
182.C.17.13 |
blocked-local, "Mac cannot host SyncNet" |
not-started; a throughput/hosting decision taken from 182.C.17.12 numbers |
182.C.18.02 |
blocked-local, aligner needs GPU worker |
done; CTC aligner on this CPU, calibrated −48.288 ms, ±32.87 ms residual (inside ITU, outside ATSC) |
182.C.18.03 |
not covered | done; GOP from the same posteriors, saturated on clean speech, threshold provisional pending 182.C.33 |
182.C.20.01–.08 |
blocked-local, "No GPU host" |
not-started; depends on 182.C.17 and on real generated active-scene clips (spend, consent-bound) |
182.C.32.01–.02 |
blocked-local, "GPU-bound research" |
not-started; a benchmark-harness definition and a licence review need no GPU |
182.C.32.03–.08 |
blocked-local, "GPU-bound research" |
blocked-local, now justified: self-hosted diffusion video generators need CUDA; none here |
Where the weights are#
Staged outside every worktree, so a scratchpad clean-up cannot delete them:
~/.cache/oshun-human-video/ — yunet.onnx (8f2383e4…), sface.onnx
(0ba9fbfa…), face_landmarker.task (64184e22…), syncnet_v2.model
(961e8696…), SyncNetModel.py (41b85aed…), LICENSE.md. 93 MB total.
Nothing under that directory is committed; 182.C.17.02 re-fetches by digest.
The evaluator interpreter is ~/.cache/oshun-human-video/venv — CPU torch,
transformers, scipy, python_speech_features. It is the single interpreter
every evaluator script runs under, including the aligner, and the specs that
shell out to those scripts skip visibly when it is absent rather than passing
quietly.
Reproducing#
The alignment and calibration scripts are now committed and rerun the numbers above from the committed fixture:
tools/human-video/evaluator/ctc_align.py --audio known-onset-clip-16k.wav --transcript "engine ships tuesday costs forty dollars"
tools/human-video/evaluator/calibrate_aligner.py --audio known-onset-clip-16k.wav --truth synth-truth.json --out aligner-calibration.json
The rest lived in the session scratchpad and is reproduced from the numbers
above: SyncNet forward pass with random tensors of the shapes listed; Kokoro
per-word synthesis with 600 ms gaps; faster-whisper with
word_timestamps=True, beam_size=5, compute_type=int8. No provider spend.