# Object Detection Provider — V1-P2-0212

## Decision

**Use ONNX YOLOv8 (`n` and `s` variants) as the default real-time provider,
RT-DETR ONNX as the higher-accuracy tier, and reserve cloud Vision APIs (Google
Vision / AWS Rekognition / Azure CV) for the "detect-rare-class" flow where
local model coverage is insufficient.** All variants load through the shared ML
runtime (V1-P2-0210); switching between them is a runtime parameter, not a
build-time choice.

---

## Constraints

1. **Real-time on consumer hardware.** ≥ 25 FPS at 640×640 on a mid-tier laptop
   without dedicated GPU, ≥ 60 FPS on WebGPU.
2. **80 COCO classes are the minimum.** Long tail (rare furniture, medical,
   agricultural objects) is acceptable to defer to cloud Vision when needed.
3. **Latency floor for AR scene labelling: ≤ 40 ms/frame.**
4. **Model size budget: ≤ 25 MB for the default tier** (browser download must
   complete in < 5s on a 50 Mbps connection).

---

## Options considered

### A. ONNX YOLOv8 — chosen for default tier

- Ultralytics ships ONNX exports directly: `yolov8n.onnx` (~6 MB FP16, ~12
  ms/frame WebGPU, COCO mAP 37.3), `yolov8s.onnx` (~22 MB, ~22 ms, mAP 44.9).
- INT8 quantisation cuts size by ~3× and CPU latency by ~25%.
- Single-stage detector — predictable latency, no two-stage RPN.
- Wide community deployment; the `.onnx` is well-tested.
- Supports OBB (oriented bbox) and segmentation variants for domains that need
  them; same loader path.

### B. RT-DETR (Real-Time DETR) — chosen for accuracy tier

- DETR family: query-based detector, no NMS post-processing.
- ONNX export from PaddlePaddle's official release: `rtdetr-r18` (~20 MB) at
  COCO mAP 46.5; `rtdetr-r50` (~50 MB) at mAP 53.1.
- Beats YOLOv8x on COCO mAP at lower size; trade-off is slower WebGPU latency
  (~30–40 ms/frame for r18).
- Shipping format: `.onnx` from PaddlePaddle x2paddle pipeline, validated
  against the PyTorch reference port.

### C. Cloud Vision APIs — kept for long-tail flows

- AWS Rekognition / Google Vision label-detection covers ~10k classes vs
  COCO's 80. Useful for "what is this object" queries where the user can
  tolerate a 200–800 ms round trip.
- Per-image billing (~$1 per 1000 images at the major providers).
- We gate cloud Vision behind an explicit user opt-in flow because it sends
  pixels off-device.

### D. ONNX YOLO-NAS / YOLOv9 — deferred

- Better mAP per FLOP than YOLOv8 on paper, but the official ONNX exports still
  have rough edges (custom ops, TensorRT-only optimisations). We will revisit at
  V2 once the exports stabilise.

---

## Latency / size budget

| variant     | model size (FP16) | WebGPU latency |  WASM SIMD | COCO mAP |
| ----------- | ----------------: | -------------: | ---------: | -------: |
| YOLOv8n     |             ~6 MB |        8–12 ms |   18–25 ms |     37.3 |
| YOLOv8s     |            ~22 MB |       18–22 ms |   35–45 ms |     44.9 |
| RT-DETR-R18 |            ~20 MB |       28–35 ms |   60–80 ms |     46.5 |
| RT-DETR-R50 |            ~50 MB |       55–70 ms | 110–140 ms |     53.1 |

---

## Output schema

```ts
export interface ObjectDetection {
  /** COCO class index (0..79) or extended class. */
  readonly classId: number;
  readonly className: string;
  readonly score: number;
  readonly bbox: { x: number; y: number; w: number; h: number };
  /** Polygon mask vertices when the model is segmentation-capable. */
  readonly segmentation?: ReadonlyArray<{ x: number; y: number }>;
}
```

Identical across YOLOv8 and RT-DETR.

---

## Action items

- [ ] Pull YOLOv8n + YOLOv8s ONNX from `ultralytics` v8.x; check FP16 + INT8
      variants into the registry with SHA-256.
- [ ] Pull RT-DETR-R18 ONNX from PaddlePaddle release; verify mAP within ±0.5 of
      the published number on COCO val.
- [ ] Wire the `libs/shared/src/ml/object-detection.ts` namespace.
- [ ] Implement client-side NMS (Rust kernel under
      `libs/shared/crates/ml-postprocess/`) for YOLOv8 (RT-DETR doesn't need
      it).

References:

- [Ultralytics YOLOv8](https://github.com/ultralytics/ultralytics)
- [RT-DETR](https://github.com/lyuwenyu/RT-DETR)
- [COCO eval](https://cocodataset.org/#detection-eval)
