Eve library

On-Device ML Runtime Strategy — V1-P2-0210

Guide

Revision 1

On this page

Decision#

Primary runtime: ONNX Runtime Web (ort-web) for browser, ONNX Runtime Node (onnxruntime-node) for server, with a unified loader at libs/shared/src/ml/runtime.ts. Reserve Rust ort (compiled to WASM or invoked through napi-rs bindings) for compute-bound, latency- critical kernels where ort-web's JS dispatch overhead dominates.

This decision is binding for V1; revisit at V2 when WebGPU support in ort-web has stabilised across all browser targets and TF.js's WebGPU backend has reached parity with ort-web's WebNN / WebGPU execution providers.


Constraints we are optimising for#

  1. Single import surface. Application code should depend on @oshun/shared/ml and not have to know whether the model is running in a browser tab, a Node serverless function, or a Rust WASM module. The shared loader exposes loadModel(uri) → InferenceSession and session.run(feeds), identical across runtimes.
  2. Cross-platform model assets. The same .onnx file must load on browser and server. Model conversion / quantisation pipelines stay in Python (PyTorch → ONNX → onnxruntime tools); the deployed artefact is one .onnx per model.
  3. Hardware-accelerated where possible, predictable everywhere. Each runtime should pick the fastest execution provider it can: ort-web prefers WebGPU, falls back to WebNN, then WASM SIMD; ort-node prefers CUDA / DirectML / CoreML, falls back to CPU. The shared API hides the choice but exposes a hint (session.metadata.executionProvider) for telemetry.
  4. Bundle size. Browser bundles must not bloat. ort-web's WASM payload is ~5 MB gzipped — acceptable when amortised across multiple models. TF.js's core + WebGL backend is similar; TF.js's WebGPU backend adds another ~1 MB.
  5. Perf parity within ±10% across runtimes for our model set. Pose estimation (MoveNet / RTMPose), object detection (YOLOv8s), ASR (Whisper-small), speaker-ID (ECAPA-TDNN), and the cross- encoder reranker all have published ONNX exports with verified numerical equivalence to their PyTorch / TF originals.

Options considered#

A. ONNX Runtime Web (ort-web) + ONNX Runtime Node — chosen#

Pros

  • Identical model file across browser and Node — no per-runtime format. PyTorch / Hugging Face exporters write ONNX natively.
  • Microsoft maintains both packages; releases are versioned in lock-step.
  • Best WebGPU support of any JS runtime as of 2024-Q4: ort-web 1.18+ ships a webgpu execution provider that lights up on Chrome unflagged and Safari behind a flag, with fallback to WebNN (Edge / mobile Chrome) and finally WASM SIMD.
  • Native Node bindings via onnxruntime-node provide CUDA / TensorRT / DirectML / CoreML execution providers — same API as ort-web.
  • Operator coverage is the broadest of any portable runtime — including the modern attention / RoPE / FlashAttention exports we need for Whisper and the BGE rerankers.
  • Shipping format is a single .onnx (or .onnx + external-data blob for >2 GB models). Tooling for INT8 / INT4 / FP16 quantisation is first-class via onnxruntime-tools.

Cons

  • Bundle size: 5 MB gzipped for the WASM payload. Mitigated by lazy-loading via the shared loader and by serving from a CDN with strong cache headers.
  • Limited mobile-WebGPU support today. We accept the WASM-SIMD fallback path for Safari iOS and older Chromium; pose / detection models that demand GPU acceleration will gate on a feature-detect in the loader.
  • Some bleeding-edge ops (FlashAttention v2, paged-KV-cache) are CUDA-only on the Node side. Browser side is one release behind.

B. TensorFlow.js + tfjs-node — rejected#

Pros

  • Smallest browser bundle when you stick to the WebGL backend (~1.5 MB gzipped).
  • TFLite ports for many mobile-optimised models exist.

Cons

  • Model format split: TF.js has its own JSON+sharded-binary format, and converting from PyTorch requires an onnx → tf → tfjs round trip that frequently breaks on attention / dynamic-shape models.
  • WebGPU backend (@tensorflow/tfjs-backend-webgpu) is still behind ort-web in operator coverage, especially for transformer attention.
  • Quantisation tooling for TF.js is ONNX-shaped anyway — the industry's centre of gravity is ONNX.
  • TF.js ecosystem activity has slowed since 2023; ort-web's release cadence is faster.

C. Rust ort via WASM — partial adoption#

Pros

  • Identical Rust binding to the same ONNX Runtime C++ core that ort-web wraps. Zero numerical drift versus ort-web / ort-node.
  • We can compile a small set of Rust kernels (audio resampling, MFCC, preprocessing) into a WASM module that integrates with ort-web's buffer types via shared Float32Arrays, avoiding double-copy.
  • Useful when we own the kernel and want SIMD-128 deterministic output (e.g., the QSAR molecular fingerprint hash).

Cons

  • For inference itself, a Rust+ort WASM build is the same ort-web binary — there's no perf win unless we replace WASM with a native binding.
  • Native bindings (napi-rs → ort Rust crate) only help on the server. The cross-runtime parity story comes from ONNX, not from Rust.

Decision: use Rust ort only for performance-critical preprocessing / postprocessing (audio resampling, image decoding, NMS for detection, beam search for ASR) where we want deterministic numerics and SIMD. The model itself runs through ort-web / ort-node.

D. Cloud-only inference — rejected for V1#

Not viable for two product surfaces:

  • Privacy-sensitive flows (Iris memory, A2A signing, on-device voice) require zero plaintext leaving the device.
  • Latency floor: pose estimation at 30 FPS cannot tolerate a round-trip; ASR partial transcripts must surface inside 200 ms.

We will still use cloud inference where the model is too large for the device (Sora-class video, frontier LLMs); those go through the existing @aje/agents API integrations.


Architecture#

Shared loader contract#

ts
// libs/shared/src/ml/runtime.ts
export interface InferenceSession {
  readonly executionProvider:
    | 'webgpu'
    | 'webnn'
    | 'wasm'
    | 'cuda'
    | 'directml'
    | 'coreml'
    | 'cpu';
  run(feeds: Record<string, Tensor>): Promise<Record<string, Tensor>>;
  release(): void;
}

export interface LoadOptions {
  readonly preferredProviders?: ReadonlyArray<
    InferenceSession['executionProvider']
  >;
  readonly graphOptimizationLevel?: 'disabled' | 'basic' | 'extended' | 'all';
  readonly enableMemPattern?: boolean;
  readonly enableCpuMemArena?: boolean;
}

export function loadModel(
  uri: string | URL | ArrayBuffer,
  options?: LoadOptions
): Promise<InferenceSession>;

Implementation switches on typeof window + typeof process:

  • Browser → import('onnxruntime-web') lazy-loaded with WASM artefact URL configured to point at our model registry CDN.
  • Node → import('onnxruntime-node') resolved via package exports; uses CUDA when CUDA_VISIBLE_DEVICES is set, otherwise CPU.

The loader feature-detects navigator.gpu for WebGPU, navigator.ml.createContext() for WebNN, falls back to WASM SIMD.

Tensor primitive#

Tensor is a thin wrapper over Float32Array / Int64Array / Uint8Array with a dims array. Both ort-web and ort-node accept this shape directly via their Tensor constructors so the loader does not copy.

Model fetching#

Models are stored at the registry described in V1-P2-0218 with SHA-256 content-addressing. The loader caches by (uri, sha256) in Cache (browser, via caches.open('oshun-ml-v1')) and in os.tmpdir() (Node). A subsequent loadModel(uri) skips both the download and the WASM compile step on cache hit.


Risk + mitigation#

risk mitigation
WebGPU adoption gap on mobile Safari preferredProviders defaults negotiate to WASM SIMD with Float16 quantisation
ort-web bundle size bloats first-paint Lazy-load via dynamic import(); loader is itself ~3 KB gzipped
ONNX op coverage lags PyTorch's bleeding edge We pin model exports to ONNX opset 17+; CI runs the export pipeline against tests
Server cold-start cost on Lambda / Cloud Run Pre-bake the WASM artefact into the container image; ort-node loads in ~50 ms
Numerical drift between ort-web and ort-node Quantisation policy fixed in onnxruntime-tools; CI runs a 1k-input KAT round trip

Action items#

  • Create libs/shared/src/ml/runtime.ts (V1-P2-0217).
  • Stand up the model registry at infra/models/ (V1-P2-0218).
  • Wire the WASM/WebGPU benchmarks (V1-P2-0219).
  • Convert each shipping model to ONNX with onnxruntime-tools and check the resulting .onnx into the registry with its SHA-256.
  • Document the cache-bust protocol when re-quantising a model.

References: