Eve library

Pose Estimation Provider — V1-P2-0211

Guide

Revision 1

On this page

Decision#

Use ONNX MoveNet (Lightning + Thunder variants) as the default provider, loaded through the shared ML runtime (V1-P2-0210, V1-P2-0217). Use RTMPose ONNX exports as the higher-accuracy option for fitness / biomechanics flows where keypoint precision matters more than throughput. MediaPipe is rejected for V1.

The two ONNX models share the runtime path, the model-registry plumbing (V1-P2-0218), and the WebGPU acceleration stack (V1-P2-0219); switching between them is a runtime parameter, not a build-time choice.


Constraints#

  1. Latency floor: ≤ 33 ms / frame for AR overlays at 30 FPS on a mid-tier laptop or current-gen mobile.
  2. Output format must include 17 COCO keypoints with per-keypoint confidence so downstream code (skeleton renderer, gesture recogniser) can branch on visibility.
  3. No model-format split. Same artefact loads on browser + Node so server-side pose extraction (e.g., uploading a workout video for analysis) and client-side AR overlays share the same code path.
  4. Privacy. Pose data should be derivable on-device; no frame should need to round-trip to a cloud endpoint for the default flow.

Options considered#

A. ONNX MoveNet (Lightning / Thunder) — chosen#

  • Lightning (192×192): ~3 MB FP16, 3–6 ms/frame on WebGPU, ~12 ms/frame on WASM SIMD. Real-time on every device that supports WebGPU; degrades gracefully on Safari iOS and old Chromium.
  • Thunder (256×256): ~7 MB FP16, ~12 ms/frame on WebGPU, ~28 ms/frame on WASM SIMD. Higher accuracy on small / occluded subjects; preferred for fitness flows.
  • 17 COCO keypoints + per-keypoint confidence — exactly the schema downstream consumers want.
  • Published ONNX exports from TensorFlow Hub with verified numerical parity to the TFLite originals.
  • Quantises to INT8 cleanly via onnxruntime-tools; INT8 Lightning ships in 1.1 MB and runs ~25% faster on CPU.

B. RTMPose ONNX (mmpose) — chosen for accuracy tier#

  • 17 COCO + 26 / 133 keypoints (Halpe / WholeBody) for finer-grained schemas.
  • ~10–15 ms/frame on WebGPU at the S size, ~40 ms at M.
  • Top-2 on the COCO-WholeBody leaderboard among real-time exports.
  • Same ONNX path as MoveNet — drop-in via the shared loader.
  • Larger artefact (15–35 MB FP16); we lazy-load on demand only when a flow opts into the higher-accuracy tier.

C. MediaPipe Pose / Tasks — rejected#

  • Excellent on-device performance but Google has migrated MediaPipe to a tasks-runtime built on TFLite + WebGL. The artefact is not ONNX, breaking our single-runtime constraint.
  • MediaPipe Tasks ships its own JS package (@mediapipe/tasks-vision) with a different API surface; bundling it doubles our ML runtime footprint.
  • Server-side use requires the C++ build, not the web build — full parity is impossible with a single .onnx artefact.
  • Output topology (33 BlazePose keypoints) is denser than COCO 17 but not interchangeable; downstream consumers would have to branch.

D. Cloud Vision pose APIs — rejected for V1#

  • AWS Rekognition / Google Vision pose estimation has a 200–800 ms round-trip floor — incompatible with 30 FPS AR overlays.
  • Per-frame billing is impractical at the throughput we need.

Numerical / latency budget#

variant model size (FP16) WebGPU latency WASM SIMD latency accuracy on COCO val (AP)
MoveNet Lightning ~3 MB 3–6 ms 10–14 ms 65.6
MoveNet Thunder ~7 MB 10–14 ms 25–32 ms 69.5
RTMPose-S 17kp ~15 MB 10–15 ms 35–45 ms 71.7
RTMPose-M WholeBody ~35 MB 30–45 ms 90–120 ms COCO-WB 75.3

Schema we expose#

ts
export interface PoseKeypoint {
  /** Index into the COCO-17 schema (0 = nose, 1 = left eye, ...). */
  readonly id: number;
  /** Pixel coordinates in the source image. */
  readonly x: number;
  readonly y: number;
  /** Per-keypoint confidence in [0, 1]. */
  readonly score: number;
  /** Optional 3D depth — populated when the model is RTMPose-3D. */
  readonly z?: number;
}

export interface PoseEstimateResult {
  readonly keypoints: ReadonlyArray<PoseKeypoint>;
  /** Per-instance bounding box in pixel coordinates. */
  readonly bbox: { x: number; y: number; w: number; h: number };
  /** Whole-instance confidence aggregate. */
  readonly score: number;
  /** Provider that produced the result, for telemetry. */
  readonly provider:
    | 'movenet-lightning'
    | 'movenet-thunder'
    | 'rtmpose-s'
    | 'rtmpose-m';
}

The output is identical between MoveNet and RTMPose so callers do not branch on provider.


Action items#

  • Convert MoveNet Lightning + Thunder TF-Hub checkpoints to ONNX via tf2onnx; verify FP32 output parity within ±1e-4 vs the original TFLite reference on the COCO val set.
  • Quantise both to INT8 and FP16; check the resulting .onnx into the registry with SHA-256.
  • Pull RTMPose-S 17kp ONNX from the official mmpose release bundle; quantise to FP16; check in.
  • Wire the pose namespace under libs/shared/src/ml/pose.ts with a single estimatePose(input, options) entrypoint that routes to the chosen variant.

References: