Decision#
Use ONNX YOLOv8 (n and s variants) as the default real-time provider,
RT-DETR ONNX as the higher-accuracy tier, and reserve cloud Vision APIs (Google
Vision / AWS Rekognition / Azure CV) for the "detect-rare-class" flow where
local model coverage is insufficient. All variants load through the shared ML
runtime (V1-P2-0210); switching between them is a runtime parameter, not a
build-time choice.
Constraints#
- Real-time on consumer hardware. ≥ 25 FPS at 640×640 on a mid-tier laptop without dedicated GPU, ≥ 60 FPS on WebGPU.
- 80 COCO classes are the minimum. Long tail (rare furniture, medical, agricultural objects) is acceptable to defer to cloud Vision when needed.
- Latency floor for AR scene labelling: ≤ 40 ms/frame.
- Model size budget: ≤ 25 MB for the default tier (browser download must complete in < 5s on a 50 Mbps connection).
Options considered#
A. ONNX YOLOv8 — chosen for default tier#
- Ultralytics ships ONNX exports directly:
yolov8n.onnx(~6 MB FP16, ~12 ms/frame WebGPU, COCO mAP 37.3),yolov8s.onnx(~22 MB, ~22 ms, mAP 44.9). - INT8 quantisation cuts size by ~3× and CPU latency by ~25%.
- Single-stage detector — predictable latency, no two-stage RPN.
- Wide community deployment; the
.onnxis well-tested. - Supports OBB (oriented bbox) and segmentation variants for domains that need them; same loader path.
B. RT-DETR (Real-Time DETR) — chosen for accuracy tier#
- DETR family: query-based detector, no NMS post-processing.
- ONNX export from PaddlePaddle's official release:
rtdetr-r18(~20 MB) at COCO mAP 46.5;rtdetr-r50(~50 MB) at mAP 53.1. - Beats YOLOv8x on COCO mAP at lower size; trade-off is slower WebGPU latency (~30–40 ms/frame for r18).
- Shipping format:
.onnxfrom PaddlePaddle x2paddle pipeline, validated against the PyTorch reference port.
C. Cloud Vision APIs — kept for long-tail flows#
- AWS Rekognition / Google Vision label-detection covers ~10k classes vs COCO's 80. Useful for "what is this object" queries where the user can tolerate a 200–800 ms round trip.
- Per-image billing (~$1 per 1000 images at the major providers).
- We gate cloud Vision behind an explicit user opt-in flow because it sends pixels off-device.
D. ONNX YOLO-NAS / YOLOv9 — deferred#
- Better mAP per FLOP than YOLOv8 on paper, but the official ONNX exports still have rough edges (custom ops, TensorRT-only optimisations). We will revisit at V2 once the exports stabilise.
Latency / size budget#
| variant | model size (FP16) | WebGPU latency | WASM SIMD | COCO mAP |
|---|---|---|---|---|
| YOLOv8n | ~6 MB | 8–12 ms | 18–25 ms | 37.3 |
| YOLOv8s | ~22 MB | 18–22 ms | 35–45 ms | 44.9 |
| RT-DETR-R18 | ~20 MB | 28–35 ms | 60–80 ms | 46.5 |
| RT-DETR-R50 | ~50 MB | 55–70 ms | 110–140 ms | 53.1 |
Output schema#
ts
export interface ObjectDetection {
/** COCO class index (0..79) or extended class. */
readonly classId: number;
readonly className: string;
readonly score: number;
readonly bbox: { x: number; y: number; w: number; h: number };
/** Polygon mask vertices when the model is segmentation-capable. */
readonly segmentation?: ReadonlyArray<{ x: number; y: number }>;
}
Identical across YOLOv8 and RT-DETR.
Action items#
- Pull YOLOv8n + YOLOv8s ONNX from
ultralyticsv8.x; check FP16 + INT8 variants into the registry with SHA-256. - Pull RT-DETR-R18 ONNX from PaddlePaddle release; verify mAP within ±0.5 of the published number on COCO val.
- Wire the
libs/shared/src/ml/object-detection.tsnamespace. - Implement client-side NMS (Rust kernel under
libs/shared/crates/ml-postprocess/) for YOLOv8 (RT-DETR doesn't need it).
References: