Decision#
Use ONNX MoveNet (Lightning + Thunder variants) as the default provider, loaded through the shared ML runtime (V1-P2-0210, V1-P2-0217). Use RTMPose ONNX exports as the higher-accuracy option for fitness / biomechanics flows where keypoint precision matters more than throughput. MediaPipe is rejected for V1.
The two ONNX models share the runtime path, the model-registry plumbing (V1-P2-0218), and the WebGPU acceleration stack (V1-P2-0219); switching between them is a runtime parameter, not a build-time choice.
Constraints#
- Latency floor: ≤ 33 ms / frame for AR overlays at 30 FPS on a mid-tier laptop or current-gen mobile.
- Output format must include 17 COCO keypoints with per-keypoint confidence so downstream code (skeleton renderer, gesture recogniser) can branch on visibility.
- No model-format split. Same artefact loads on browser + Node so server-side pose extraction (e.g., uploading a workout video for analysis) and client-side AR overlays share the same code path.
- Privacy. Pose data should be derivable on-device; no frame should need to round-trip to a cloud endpoint for the default flow.
Options considered#
A. ONNX MoveNet (Lightning / Thunder) — chosen#
- Lightning (192×192): ~3 MB FP16, 3–6 ms/frame on WebGPU, ~12 ms/frame on WASM SIMD. Real-time on every device that supports WebGPU; degrades gracefully on Safari iOS and old Chromium.
- Thunder (256×256): ~7 MB FP16, ~12 ms/frame on WebGPU, ~28 ms/frame on WASM SIMD. Higher accuracy on small / occluded subjects; preferred for fitness flows.
- 17 COCO keypoints + per-keypoint confidence — exactly the schema downstream consumers want.
- Published ONNX exports from TensorFlow Hub with verified numerical parity to the TFLite originals.
- Quantises to INT8 cleanly via
onnxruntime-tools; INT8 Lightning ships in 1.1 MB and runs ~25% faster on CPU.
B. RTMPose ONNX (mmpose) — chosen for accuracy tier#
- 17 COCO + 26 / 133 keypoints (Halpe / WholeBody) for finer-grained schemas.
- ~10–15 ms/frame on WebGPU at the
Ssize, ~40 ms atM. - Top-2 on the COCO-WholeBody leaderboard among real-time exports.
- Same ONNX path as MoveNet — drop-in via the shared loader.
- Larger artefact (15–35 MB FP16); we lazy-load on demand only when a flow opts into the higher-accuracy tier.
C. MediaPipe Pose / Tasks — rejected#
- Excellent on-device performance but Google has migrated MediaPipe to a tasks-runtime built on TFLite + WebGL. The artefact is not ONNX, breaking our single-runtime constraint.
- MediaPipe Tasks ships its own JS package (
@mediapipe/tasks-vision) with a different API surface; bundling it doubles our ML runtime footprint. - Server-side use requires the C++ build, not the web build — full parity is
impossible with a single
.onnxartefact. - Output topology (33 BlazePose keypoints) is denser than COCO 17 but not interchangeable; downstream consumers would have to branch.
D. Cloud Vision pose APIs — rejected for V1#
- AWS Rekognition / Google Vision pose estimation has a 200–800 ms round-trip floor — incompatible with 30 FPS AR overlays.
- Per-frame billing is impractical at the throughput we need.
Numerical / latency budget#
| variant | model size (FP16) | WebGPU latency | WASM SIMD latency | accuracy on COCO val (AP) |
|---|---|---|---|---|
| MoveNet Lightning | ~3 MB | 3–6 ms | 10–14 ms | 65.6 |
| MoveNet Thunder | ~7 MB | 10–14 ms | 25–32 ms | 69.5 |
| RTMPose-S 17kp | ~15 MB | 10–15 ms | 35–45 ms | 71.7 |
| RTMPose-M WholeBody | ~35 MB | 30–45 ms | 90–120 ms | COCO-WB 75.3 |
Schema we expose#
ts
export interface PoseKeypoint {
/** Index into the COCO-17 schema (0 = nose, 1 = left eye, ...). */
readonly id: number;
/** Pixel coordinates in the source image. */
readonly x: number;
readonly y: number;
/** Per-keypoint confidence in [0, 1]. */
readonly score: number;
/** Optional 3D depth — populated when the model is RTMPose-3D. */
readonly z?: number;
}
export interface PoseEstimateResult {
readonly keypoints: ReadonlyArray<PoseKeypoint>;
/** Per-instance bounding box in pixel coordinates. */
readonly bbox: { x: number; y: number; w: number; h: number };
/** Whole-instance confidence aggregate. */
readonly score: number;
/** Provider that produced the result, for telemetry. */
readonly provider:
| 'movenet-lightning'
| 'movenet-thunder'
| 'rtmpose-s'
| 'rtmpose-m';
}
The output is identical between MoveNet and RTMPose so callers do not branch on provider.
Action items#
- Convert MoveNet Lightning + Thunder TF-Hub checkpoints to ONNX via
tf2onnx; verify FP32 output parity within ±1e-4 vs the original TFLite reference on the COCO val set. - Quantise both to INT8 and FP16; check the resulting
.onnxinto the registry with SHA-256. - Pull RTMPose-S 17kp ONNX from the official mmpose release bundle; quantise to FP16; check in.
- Wire the pose namespace under
libs/shared/src/ml/pose.tswith a singleestimatePose(input, options)entrypoint that routes to the chosen variant.
References: