Eve library

Where the sync evaluators run (`182.C.17.13`)

Guide

Revision 1

On this page

Last updated: 2026-09-10

The decision this phase asked for, taken from the 182.C.17.12 numbers rather than from a preference. The measurements are in apps/oshun/bff/src/generation/human-video-evaluator-throughput.ts, kept as data so a different host can be measured the same way and compared.

The numbers the decision rests on#

Measured on the Linux dev box — 4 cores, no GPU — over real clips of a real face through the real pipeline, at two lengths to confirm linearity.

Stage 10 s clip 20 s clip Per video-second
prepare (decode/detect/crop) 7.09 s 14.99 s ~0.73 s
SyncNet v2 scoring 9.07 s 18.26 s ~0.91 s
speech activity + mouth motion 0.47 s 0.91 s ~0.05 s
total 16.63 s 34.16 s ~1.70 s

0.59× realtime, and linear: the same per-second figure at both lengths, so capacity is multiplication rather than a curve fit.

And the concurrency curve, which is the part that decided the hosting shape:

Concurrent evaluators Video-seconds per wall-second Against one
1 0.42 —
2 0.50 +19%
3 0.52 +24%
4 0.48 +14%

It turns over inside the measured range. Four evaluators — one per core — are worse than three. Each process already saturates the cores with its own threads (torch at 4, plus ffmpeg and OpenCV), so more processes contend instead of parallelising.

The decision#

CPU inside the BFF worker, one evaluator at a time, is the production path for the current shape of the workload. No GPU worker is provisioned now.

Three reasons, in order of weight.

1. The throughput is sufficient for the workload this phase produces. At 0.42 video-seconds per wall-second a ten-second take is evaluated in about 24 seconds. Generation itself takes a provider tens of seconds to minutes, so evaluation is not the long pole — it is a fraction of a step that is already slower than it is.

2. A GPU worker would buy latency the pipeline cannot spend. The evaluation happens after the render and before the release gate. Shaving twenty seconds off a step inside a multi-minute job changes nothing a creator can perceive, and it adds a network hop, a queue, a second place for weights to be stale, and a second thing to be down.

3. The scaling lever is horizontal, not vertical. The curve says a single box gains almost nothing from concurrency, which means the way to evaluate more takes is more workers rather than a bigger one — and more BFF workers are something the deployment already knows how to make.

What would reverse it#

Named now so it is a threshold rather than a judgement call later:

  • Sustained demand above one worker's capacity. 0.42 video-seconds per wall-second is about 25 video-minutes an hour per worker. Past the point where adding workers is more expensive than a GPU host, the arithmetic changes.
  • An evaluator that needs a GPU. The current two run on CPU because that was measured, not assumed. A third — 182.C.17.01 lists the candidates — could require one, and then the hosting question is answered by the model rather than by throughput.
  • Long-form takes. At 0.59× realtime a ten-minute take costs seventeen minutes of CPU. 182.C.22 is where that lands, and it is a different calculation from the shot-length work this phase measured.

The route is already in the repo when it does reverse: RunPod through libs/isis/runpod-surface/ and runpod-comfy-client.ts, which is how the GPU-bound generation lanes already reach hardware.

The CPU path is kept regardless#

Whatever hosts production, the CPU path stays, and this is not a fallback — it is what CI runs. Every measurement in 182.C.17.03 through 182.C.17.11 was produced by it on this box, including the adversarial suite. An evaluation stack that can only be exercised on hardware CI does not have is a stack whose regressions are found in production.

The scheduling policy that follows#

Enforced by assertDoesNotStarveRenders, not left to configuration:

  • One evaluator at a time on a host with four cores or fewer. Three would deliver 24% more work while taking three-quarters of the box from the renders and transcodes that are the product; that trade is only worth making where there are cores nothing else wants.
  • Three cores reserved for render, transcode and the request path.
  • Threads capped as well as processes. They are the same resource: one process with every thread starves the box exactly as thoroughly as four processes with one thread each, and a policy that counted only processes would have a hole the size of the box.