# On-Device ML Runtime Strategy — V1-P2-0210

## Decision

**Primary runtime: ONNX Runtime Web (ort-web) for browser, ONNX Runtime Node
(`onnxruntime-node`) for server, with a unified loader at
`libs/shared/src/ml/runtime.ts`. Reserve Rust `ort` (compiled to WASM or invoked
through `napi-rs` bindings) for compute-bound, latency- critical kernels where
ort-web's JS dispatch overhead dominates.**

This decision is binding for V1; revisit at V2 when WebGPU support in ort-web
has stabilised across all browser targets and TF.js's WebGPU backend has reached
parity with ort-web's WebNN / WebGPU execution providers.

---

## Constraints we are optimising for

1. **Single import surface.** Application code should depend on
   `@oshun/shared/ml` and not have to know whether the model is running in a
   browser tab, a Node serverless function, or a Rust WASM module. The shared
   loader exposes `loadModel(uri) → InferenceSession` and `session.run(feeds)`,
   identical across runtimes.
2. **Cross-platform model assets.** The same `.onnx` file must load on browser
   and server. Model conversion / quantisation pipelines stay in Python (PyTorch
   → ONNX → onnxruntime tools); the deployed artefact is one `.onnx` per model.
3. **Hardware-accelerated where possible, predictable everywhere.** Each runtime
   should pick the fastest execution provider it can: ort-web prefers WebGPU,
   falls back to WebNN, then WASM SIMD; ort-node prefers CUDA / DirectML /
   CoreML, falls back to CPU. The shared API hides the choice but exposes a hint
   (`session.metadata.executionProvider`) for telemetry.
4. **Bundle size.** Browser bundles must not bloat. ort-web's WASM payload is ~5
   MB gzipped — acceptable when amortised across multiple models. TF.js's core +
   WebGL backend is similar; TF.js's WebGPU backend adds another ~1 MB.
5. **Perf parity within ±10% across runtimes for our model set.** Pose
   estimation (MoveNet / RTMPose), object detection (YOLOv8s), ASR
   (Whisper-small), speaker-ID (ECAPA-TDNN), and the cross- encoder reranker all
   have published ONNX exports with verified numerical equivalence to their
   PyTorch / TF originals.

---

## Options considered

### A. ONNX Runtime Web (ort-web) + ONNX Runtime Node — chosen

**Pros**

- Identical model file across browser and Node — no per-runtime format. PyTorch
  / Hugging Face exporters write ONNX natively.
- Microsoft maintains both packages; releases are versioned in lock-step.
- Best WebGPU support of any JS runtime as of 2024-Q4: ort-web 1.18+ ships a
  `webgpu` execution provider that lights up on Chrome unflagged and Safari
  behind a flag, with fallback to WebNN (Edge / mobile Chrome) and finally WASM
  SIMD.
- Native Node bindings via `onnxruntime-node` provide CUDA / TensorRT / DirectML
  / CoreML execution providers — same API as ort-web.
- Operator coverage is the broadest of any portable runtime — including the
  modern attention / RoPE / FlashAttention exports we need for Whisper and the
  BGE rerankers.
- Shipping format is a single `.onnx` (or `.onnx` + external-data blob for >2 GB
  models). Tooling for INT8 / INT4 / FP16 quantisation is first-class via
  `onnxruntime-tools`.

**Cons**

- Bundle size: 5 MB gzipped for the WASM payload. Mitigated by lazy-loading via
  the shared loader and by serving from a CDN with strong cache headers.
- Limited mobile-WebGPU support today. We accept the WASM-SIMD fallback path for
  Safari iOS and older Chromium; pose / detection models that demand GPU
  acceleration will gate on a feature-detect in the loader.
- Some bleeding-edge ops (FlashAttention v2, paged-KV-cache) are CUDA-only on
  the Node side. Browser side is one release behind.

### B. TensorFlow.js + tfjs-node — rejected

**Pros**

- Smallest browser bundle when you stick to the WebGL backend (~1.5 MB gzipped).
- TFLite ports for many mobile-optimised models exist.

**Cons**

- Model format split: TF.js has its own JSON+sharded-binary format, and
  converting from PyTorch requires an `onnx → tf → tfjs` round trip that
  frequently breaks on attention / dynamic-shape models.
- WebGPU backend (`@tensorflow/tfjs-backend-webgpu`) is still behind ort-web in
  operator coverage, especially for transformer attention.
- Quantisation tooling for TF.js is ONNX-shaped _anyway_ — the industry's centre
  of gravity is ONNX.
- TF.js ecosystem activity has slowed since 2023; ort-web's release cadence is
  faster.

### C. Rust `ort` via WASM — partial adoption

**Pros**

- Identical Rust binding to the same ONNX Runtime C++ core that ort-web wraps.
  Zero numerical drift versus ort-web / ort-node.
- We can compile a small set of Rust kernels (audio resampling, MFCC,
  preprocessing) into a WASM module that integrates with ort-web's buffer types
  via shared `Float32Array`s, avoiding double-copy.
- Useful when we own the kernel and want SIMD-128 deterministic output (e.g.,
  the QSAR molecular fingerprint hash).

**Cons**

- For inference itself, a Rust+ort WASM build is _the same_ ort-web binary —
  there's no perf win unless we replace WASM with a native binding.
- Native bindings (`napi-rs` → `ort` Rust crate) only help on the server. The
  cross-runtime parity story comes from ONNX, not from Rust.

**Decision:** use Rust `ort` only for performance-critical preprocessing /
postprocessing (audio resampling, image decoding, NMS for detection, beam search
for ASR) where we want deterministic numerics and SIMD. The model itself runs
through ort-web / ort-node.

### D. Cloud-only inference — rejected for V1

Not viable for two product surfaces:

- **Privacy-sensitive flows** (Iris memory, A2A signing, on-device voice)
  require zero plaintext leaving the device.
- **Latency floor**: pose estimation at 30 FPS cannot tolerate a round-trip; ASR
  partial transcripts must surface inside 200 ms.

We will still use cloud inference where the model is too large for the device
(Sora-class video, frontier LLMs); those go through the existing `@aje/agents`
API integrations.

---

## Architecture

### Shared loader contract

```ts
// libs/shared/src/ml/runtime.ts
export interface InferenceSession {
  readonly executionProvider:
    | 'webgpu'
    | 'webnn'
    | 'wasm'
    | 'cuda'
    | 'directml'
    | 'coreml'
    | 'cpu';
  run(feeds: Record<string, Tensor>): Promise<Record<string, Tensor>>;
  release(): void;
}

export interface LoadOptions {
  readonly preferredProviders?: ReadonlyArray<
    InferenceSession['executionProvider']
  >;
  readonly graphOptimizationLevel?: 'disabled' | 'basic' | 'extended' | 'all';
  readonly enableMemPattern?: boolean;
  readonly enableCpuMemArena?: boolean;
}

export function loadModel(
  uri: string | URL | ArrayBuffer,
  options?: LoadOptions
): Promise<InferenceSession>;
```

Implementation switches on `typeof window` + `typeof process`:

- Browser → `import('onnxruntime-web')` lazy-loaded with WASM artefact URL
  configured to point at our model registry CDN.
- Node → `import('onnxruntime-node')` resolved via package exports; uses CUDA
  when `CUDA_VISIBLE_DEVICES` is set, otherwise CPU.

The loader feature-detects `navigator.gpu` for WebGPU,
`navigator.ml.createContext()` for WebNN, falls back to WASM SIMD.

### Tensor primitive

`Tensor` is a thin wrapper over `Float32Array` / `Int64Array` / `Uint8Array`
with a `dims` array. Both ort-web and ort-node accept this shape directly via
their `Tensor` constructors so the loader does not copy.

### Model fetching

Models are stored at the registry described in V1-P2-0218 with SHA-256
content-addressing. The loader caches by `(uri, sha256)` in `Cache` (browser,
via `caches.open('oshun-ml-v1')`) and in `os.tmpdir()` (Node). A subsequent
`loadModel(uri)` skips both the download and the WASM compile step on cache hit.

---

## Risk + mitigation

| risk                                          | mitigation                                                                          |
| --------------------------------------------- | ----------------------------------------------------------------------------------- |
| WebGPU adoption gap on mobile Safari          | `preferredProviders` defaults negotiate to WASM SIMD with `Float16` quantisation    |
| ort-web bundle size bloats first-paint        | Lazy-load via dynamic `import()`; loader is itself ~3 KB gzipped                    |
| ONNX op coverage lags PyTorch's bleeding edge | We pin model exports to ONNX opset 17+; CI runs the export pipeline against tests   |
| Server cold-start cost on Lambda / Cloud Run  | Pre-bake the WASM artefact into the container image; ort-node loads in ~50 ms       |
| Numerical drift between ort-web and ort-node  | Quantisation policy fixed in `onnxruntime-tools`; CI runs a 1k-input KAT round trip |

---

## Action items

- [ ] Create `libs/shared/src/ml/runtime.ts` (V1-P2-0217).
- [ ] Stand up the model registry at `infra/models/` (V1-P2-0218).
- [ ] Wire the WASM/WebGPU benchmarks (V1-P2-0219).
- [ ] Convert each shipping model to ONNX with `onnxruntime-tools` and check the
      resulting `.onnx` into the registry with its SHA-256.
- [ ] Document the cache-bust protocol when re-quantising a model.

References:

- [ONNX Runtime Web docs](https://onnxruntime.ai/docs/get-started/with-javascript/web.html)
- [ONNX Runtime Node docs](https://onnxruntime.ai/docs/get-started/with-javascript/node.html)
- [ort Rust crate](https://docs.rs/ort/latest/ort/)
- [WebGPU spec — Compute pipeline](https://www.w3.org/TR/webgpu/#compute-pipeline)
