Decision#
Primary runtime: ONNX Runtime Web (ort-web) for browser, ONNX Runtime Node
(onnxruntime-node) for server, with a unified loader at
libs/shared/src/ml/runtime.ts. Reserve Rust ort (compiled to WASM or invoked
through napi-rs bindings) for compute-bound, latency- critical kernels where
ort-web's JS dispatch overhead dominates.
This decision is binding for V1; revisit at V2 when WebGPU support in ort-web has stabilised across all browser targets and TF.js's WebGPU backend has reached parity with ort-web's WebNN / WebGPU execution providers.
Constraints we are optimising for#
- Single import surface. Application code should depend on
@oshun/shared/mland not have to know whether the model is running in a browser tab, a Node serverless function, or a Rust WASM module. The shared loader exposesloadModel(uri) → InferenceSessionandsession.run(feeds), identical across runtimes. - Cross-platform model assets. The same
.onnxfile must load on browser and server. Model conversion / quantisation pipelines stay in Python (PyTorch → ONNX → onnxruntime tools); the deployed artefact is one.onnxper model. - Hardware-accelerated where possible, predictable everywhere. Each runtime
should pick the fastest execution provider it can: ort-web prefers WebGPU,
falls back to WebNN, then WASM SIMD; ort-node prefers CUDA / DirectML /
CoreML, falls back to CPU. The shared API hides the choice but exposes a hint
(
session.metadata.executionProvider) for telemetry. - Bundle size. Browser bundles must not bloat. ort-web's WASM payload is ~5 MB gzipped — acceptable when amortised across multiple models. TF.js's core + WebGL backend is similar; TF.js's WebGPU backend adds another ~1 MB.
- Perf parity within ±10% across runtimes for our model set. Pose estimation (MoveNet / RTMPose), object detection (YOLOv8s), ASR (Whisper-small), speaker-ID (ECAPA-TDNN), and the cross- encoder reranker all have published ONNX exports with verified numerical equivalence to their PyTorch / TF originals.
Options considered#
A. ONNX Runtime Web (ort-web) + ONNX Runtime Node — chosen#
Pros
- Identical model file across browser and Node — no per-runtime format. PyTorch / Hugging Face exporters write ONNX natively.
- Microsoft maintains both packages; releases are versioned in lock-step.
- Best WebGPU support of any JS runtime as of 2024-Q4: ort-web 1.18+ ships a
webgpuexecution provider that lights up on Chrome unflagged and Safari behind a flag, with fallback to WebNN (Edge / mobile Chrome) and finally WASM SIMD. - Native Node bindings via
onnxruntime-nodeprovide CUDA / TensorRT / DirectML / CoreML execution providers — same API as ort-web. - Operator coverage is the broadest of any portable runtime — including the modern attention / RoPE / FlashAttention exports we need for Whisper and the BGE rerankers.
- Shipping format is a single
.onnx(or.onnx+ external-data blob for >2 GB models). Tooling for INT8 / INT4 / FP16 quantisation is first-class viaonnxruntime-tools.
Cons
- Bundle size: 5 MB gzipped for the WASM payload. Mitigated by lazy-loading via the shared loader and by serving from a CDN with strong cache headers.
- Limited mobile-WebGPU support today. We accept the WASM-SIMD fallback path for Safari iOS and older Chromium; pose / detection models that demand GPU acceleration will gate on a feature-detect in the loader.
- Some bleeding-edge ops (FlashAttention v2, paged-KV-cache) are CUDA-only on the Node side. Browser side is one release behind.
B. TensorFlow.js + tfjs-node — rejected#
Pros
- Smallest browser bundle when you stick to the WebGL backend (~1.5 MB gzipped).
- TFLite ports for many mobile-optimised models exist.
Cons
- Model format split: TF.js has its own JSON+sharded-binary format, and
converting from PyTorch requires an
onnx → tf → tfjsround trip that frequently breaks on attention / dynamic-shape models. - WebGPU backend (
@tensorflow/tfjs-backend-webgpu) is still behind ort-web in operator coverage, especially for transformer attention. - Quantisation tooling for TF.js is ONNX-shaped anyway — the industry's centre of gravity is ONNX.
- TF.js ecosystem activity has slowed since 2023; ort-web's release cadence is faster.
C. Rust ort via WASM — partial adoption#
Pros
- Identical Rust binding to the same ONNX Runtime C++ core that ort-web wraps. Zero numerical drift versus ort-web / ort-node.
- We can compile a small set of Rust kernels (audio resampling, MFCC,
preprocessing) into a WASM module that integrates with ort-web's buffer types
via shared
Float32Arrays, avoiding double-copy. - Useful when we own the kernel and want SIMD-128 deterministic output (e.g., the QSAR molecular fingerprint hash).
Cons
- For inference itself, a Rust+ort WASM build is the same ort-web binary — there's no perf win unless we replace WASM with a native binding.
- Native bindings (
napi-rs→ortRust crate) only help on the server. The cross-runtime parity story comes from ONNX, not from Rust.
Decision: use Rust ort only for performance-critical preprocessing /
postprocessing (audio resampling, image decoding, NMS for detection, beam search
for ASR) where we want deterministic numerics and SIMD. The model itself runs
through ort-web / ort-node.
D. Cloud-only inference — rejected for V1#
Not viable for two product surfaces:
- Privacy-sensitive flows (Iris memory, A2A signing, on-device voice) require zero plaintext leaving the device.
- Latency floor: pose estimation at 30 FPS cannot tolerate a round-trip; ASR partial transcripts must surface inside 200 ms.
We will still use cloud inference where the model is too large for the device
(Sora-class video, frontier LLMs); those go through the existing @aje/agents
API integrations.
Architecture#
Shared loader contract#
// libs/shared/src/ml/runtime.ts
export interface InferenceSession {
readonly executionProvider:
| 'webgpu'
| 'webnn'
| 'wasm'
| 'cuda'
| 'directml'
| 'coreml'
| 'cpu';
run(feeds: Record<string, Tensor>): Promise<Record<string, Tensor>>;
release(): void;
}
export interface LoadOptions {
readonly preferredProviders?: ReadonlyArray<
InferenceSession['executionProvider']
>;
readonly graphOptimizationLevel?: 'disabled' | 'basic' | 'extended' | 'all';
readonly enableMemPattern?: boolean;
readonly enableCpuMemArena?: boolean;
}
export function loadModel(
uri: string | URL | ArrayBuffer,
options?: LoadOptions
): Promise<InferenceSession>;
Implementation switches on typeof window + typeof process:
- Browser →
import('onnxruntime-web')lazy-loaded with WASM artefact URL configured to point at our model registry CDN. - Node →
import('onnxruntime-node')resolved via package exports; uses CUDA whenCUDA_VISIBLE_DEVICESis set, otherwise CPU.
The loader feature-detects navigator.gpu for WebGPU,
navigator.ml.createContext() for WebNN, falls back to WASM SIMD.
Tensor primitive#
Tensor is a thin wrapper over Float32Array / Int64Array / Uint8Array
with a dims array. Both ort-web and ort-node accept this shape directly via
their Tensor constructors so the loader does not copy.
Model fetching#
Models are stored at the registry described in V1-P2-0218 with SHA-256
content-addressing. The loader caches by (uri, sha256) in Cache (browser,
via caches.open('oshun-ml-v1')) and in os.tmpdir() (Node). A subsequent
loadModel(uri) skips both the download and the WASM compile step on cache hit.
Risk + mitigation#
| risk | mitigation |
|---|---|
| WebGPU adoption gap on mobile Safari | preferredProviders defaults negotiate to WASM SIMD with Float16 quantisation |
| ort-web bundle size bloats first-paint | Lazy-load via dynamic import(); loader is itself ~3 KB gzipped |
| ONNX op coverage lags PyTorch's bleeding edge | We pin model exports to ONNX opset 17+; CI runs the export pipeline against tests |
| Server cold-start cost on Lambda / Cloud Run | Pre-bake the WASM artefact into the container image; ort-node loads in ~50 ms |
| Numerical drift between ort-web and ort-node | Quantisation policy fixed in onnxruntime-tools; CI runs a 1k-input KAT round trip |
Action items#
- Create
libs/shared/src/ml/runtime.ts(V1-P2-0217). - Stand up the model registry at
infra/models/(V1-P2-0218). - Wire the WASM/WebGPU benchmarks (V1-P2-0219).
- Convert each shipping model to ONNX with
onnxruntime-toolsand check the resulting.onnxinto the registry with its SHA-256. - Document the cache-bust protocol when re-quantising a model.
References: