Eve library

Object Detection Provider — V1-P2-0212

Guide

Revision 1

On this page

Decision#

Use ONNX YOLOv8 (n and s variants) as the default real-time provider, RT-DETR ONNX as the higher-accuracy tier, and reserve cloud Vision APIs (Google Vision / AWS Rekognition / Azure CV) for the "detect-rare-class" flow where local model coverage is insufficient. All variants load through the shared ML runtime (V1-P2-0210); switching between them is a runtime parameter, not a build-time choice.


Constraints#

  1. Real-time on consumer hardware. ≥ 25 FPS at 640×640 on a mid-tier laptop without dedicated GPU, ≥ 60 FPS on WebGPU.
  2. 80 COCO classes are the minimum. Long tail (rare furniture, medical, agricultural objects) is acceptable to defer to cloud Vision when needed.
  3. Latency floor for AR scene labelling: ≤ 40 ms/frame.
  4. Model size budget: ≤ 25 MB for the default tier (browser download must complete in < 5s on a 50 Mbps connection).

Options considered#

A. ONNX YOLOv8 — chosen for default tier#

  • Ultralytics ships ONNX exports directly: yolov8n.onnx (~6 MB FP16, ~12 ms/frame WebGPU, COCO mAP 37.3), yolov8s.onnx (~22 MB, ~22 ms, mAP 44.9).
  • INT8 quantisation cuts size by ~3× and CPU latency by ~25%.
  • Single-stage detector — predictable latency, no two-stage RPN.
  • Wide community deployment; the .onnx is well-tested.
  • Supports OBB (oriented bbox) and segmentation variants for domains that need them; same loader path.

B. RT-DETR (Real-Time DETR) — chosen for accuracy tier#

  • DETR family: query-based detector, no NMS post-processing.
  • ONNX export from PaddlePaddle's official release: rtdetr-r18 (~20 MB) at COCO mAP 46.5; rtdetr-r50 (~50 MB) at mAP 53.1.
  • Beats YOLOv8x on COCO mAP at lower size; trade-off is slower WebGPU latency (~30–40 ms/frame for r18).
  • Shipping format: .onnx from PaddlePaddle x2paddle pipeline, validated against the PyTorch reference port.

C. Cloud Vision APIs — kept for long-tail flows#

  • AWS Rekognition / Google Vision label-detection covers ~10k classes vs COCO's 80. Useful for "what is this object" queries where the user can tolerate a 200–800 ms round trip.
  • Per-image billing (~$1 per 1000 images at the major providers).
  • We gate cloud Vision behind an explicit user opt-in flow because it sends pixels off-device.

D. ONNX YOLO-NAS / YOLOv9 — deferred#

  • Better mAP per FLOP than YOLOv8 on paper, but the official ONNX exports still have rough edges (custom ops, TensorRT-only optimisations). We will revisit at V2 once the exports stabilise.

Latency / size budget#

variant model size (FP16) WebGPU latency WASM SIMD COCO mAP
YOLOv8n ~6 MB 8–12 ms 18–25 ms 37.3
YOLOv8s ~22 MB 18–22 ms 35–45 ms 44.9
RT-DETR-R18 ~20 MB 28–35 ms 60–80 ms 46.5
RT-DETR-R50 ~50 MB 55–70 ms 110–140 ms 53.1

Output schema#

ts
export interface ObjectDetection {
  /** COCO class index (0..79) or extended class. */
  readonly classId: number;
  readonly className: string;
  readonly score: number;
  readonly bbox: { x: number; y: number; w: number; h: number };
  /** Polygon mask vertices when the model is segmentation-capable. */
  readonly segmentation?: ReadonlyArray<{ x: number; y: number }>;
}

Identical across YOLOv8 and RT-DETR.


Action items#

  • Pull YOLOv8n + YOLOv8s ONNX from ultralytics v8.x; check FP16 + INT8 variants into the registry with SHA-256.
  • Pull RT-DETR-R18 ONNX from PaddlePaddle release; verify mAP within ±0.5 of the published number on COCO val.
  • Wire the libs/shared/src/ml/object-detection.ts namespace.
  • Implement client-side NMS (Rust kernel under libs/shared/crates/ml-postprocess/) for YOLOv8 (RT-DETR doesn't need it).

References: