# Pose Estimation Provider — V1-P2-0211

## Decision

**Use ONNX MoveNet (Lightning + Thunder variants) as the default provider,
loaded through the shared ML runtime (V1-P2-0210, V1-P2-0217). Use RTMPose ONNX
exports as the higher-accuracy option for fitness / biomechanics flows where
keypoint precision matters more than throughput. MediaPipe is rejected for V1.**

The two ONNX models share the runtime path, the model-registry plumbing
(V1-P2-0218), and the WebGPU acceleration stack (V1-P2-0219); switching between
them is a runtime parameter, not a build-time choice.

---

## Constraints

1. **Latency floor: ≤ 33 ms / frame** for AR overlays at 30 FPS on a mid-tier
   laptop or current-gen mobile.
2. **Output format must include 17 COCO keypoints** with per-keypoint confidence
   so downstream code (skeleton renderer, gesture recogniser) can branch on
   visibility.
3. **No model-format split.** Same artefact loads on browser + Node so
   server-side pose extraction (e.g., uploading a workout video for analysis)
   and client-side AR overlays share the same code path.
4. **Privacy.** Pose data should be derivable on-device; no frame should need to
   round-trip to a cloud endpoint for the default flow.

---

## Options considered

### A. ONNX MoveNet (Lightning / Thunder) — chosen

- **Lightning (192×192)**: ~3 MB FP16, 3–6 ms/frame on WebGPU, ~12 ms/frame on
  WASM SIMD. Real-time on every device that supports WebGPU; degrades gracefully
  on Safari iOS and old Chromium.
- **Thunder (256×256)**: ~7 MB FP16, ~12 ms/frame on WebGPU, ~28 ms/frame on
  WASM SIMD. Higher accuracy on small / occluded subjects; preferred for fitness
  flows.
- 17 COCO keypoints + per-keypoint confidence — exactly the schema downstream
  consumers want.
- Published ONNX exports from TensorFlow Hub with verified numerical parity to
  the TFLite originals.
- Quantises to INT8 cleanly via `onnxruntime-tools`; INT8 Lightning ships in 1.1
  MB and runs ~25% faster on CPU.

### B. RTMPose ONNX (mmpose) — chosen for accuracy tier

- 17 COCO + 26 / 133 keypoints (Halpe / WholeBody) for finer-grained schemas.
- ~10–15 ms/frame on WebGPU at the `S` size, ~40 ms at `M`.
- Top-2 on the COCO-WholeBody leaderboard among real-time exports.
- Same ONNX path as MoveNet — drop-in via the shared loader.
- Larger artefact (15–35 MB FP16); we lazy-load on demand only when a flow opts
  into the higher-accuracy tier.

### C. MediaPipe Pose / Tasks — rejected

- Excellent on-device performance but Google has migrated MediaPipe to a
  tasks-runtime built on TFLite + WebGL. The artefact is _not_ ONNX, breaking
  our single-runtime constraint.
- MediaPipe Tasks ships its own JS package (`@mediapipe/tasks-vision`) with a
  different API surface; bundling it doubles our ML runtime footprint.
- Server-side use requires the C++ build, not the web build — full parity is
  impossible with a single `.onnx` artefact.
- Output topology (33 BlazePose keypoints) is denser than COCO 17 but not
  interchangeable; downstream consumers would have to branch.

### D. Cloud Vision pose APIs — rejected for V1

- AWS Rekognition / Google Vision pose estimation has a 200–800 ms round-trip
  floor — incompatible with 30 FPS AR overlays.
- Per-frame billing is impractical at the throughput we need.

---

## Numerical / latency budget

| variant             | model size (FP16) | WebGPU latency | WASM SIMD latency | accuracy on COCO val (AP) |
| ------------------- | ----------------: | -------------: | ----------------: | ------------------------: |
| MoveNet Lightning   |             ~3 MB |         3–6 ms |          10–14 ms |                      65.6 |
| MoveNet Thunder     |             ~7 MB |       10–14 ms |          25–32 ms |                      69.5 |
| RTMPose-S 17kp      |            ~15 MB |       10–15 ms |          35–45 ms |                      71.7 |
| RTMPose-M WholeBody |            ~35 MB |       30–45 ms |         90–120 ms |              COCO-WB 75.3 |

---

## Schema we expose

```ts
export interface PoseKeypoint {
  /** Index into the COCO-17 schema (0 = nose, 1 = left eye, ...). */
  readonly id: number;
  /** Pixel coordinates in the source image. */
  readonly x: number;
  readonly y: number;
  /** Per-keypoint confidence in [0, 1]. */
  readonly score: number;
  /** Optional 3D depth — populated when the model is RTMPose-3D. */
  readonly z?: number;
}

export interface PoseEstimateResult {
  readonly keypoints: ReadonlyArray<PoseKeypoint>;
  /** Per-instance bounding box in pixel coordinates. */
  readonly bbox: { x: number; y: number; w: number; h: number };
  /** Whole-instance confidence aggregate. */
  readonly score: number;
  /** Provider that produced the result, for telemetry. */
  readonly provider:
    | 'movenet-lightning'
    | 'movenet-thunder'
    | 'rtmpose-s'
    | 'rtmpose-m';
}
```

The output is identical between MoveNet and RTMPose so callers do not branch on
provider.

---

## Action items

- [ ] Convert MoveNet Lightning + Thunder TF-Hub checkpoints to ONNX via
      `tf2onnx`; verify FP32 output parity within ±1e-4 vs the original TFLite
      reference on the COCO val set.
- [ ] Quantise both to INT8 and FP16; check the resulting `.onnx` into the
      registry with SHA-256.
- [ ] Pull RTMPose-S 17kp ONNX from the official mmpose release bundle; quantise
      to FP16; check in.
- [ ] Wire the pose namespace under `libs/shared/src/ml/pose.ts` with a single
      `estimatePose(input, options)` entrypoint that routes to the chosen
      variant.

References:

- [MoveNet TF Hub](https://tfhub.dev/google/movenet/multipose/lightning)
- [RTMPose / mmpose](https://github.com/open-mmlab/mmpose)
- [tf2onnx](https://github.com/onnx/tensorflow-onnx)
