# @euterpe/realtime-engine

On-device Rust engine for **Magenta RealTime 2 (MRT2)** — the SpectroStream
codec language model that drives frame-level, ~200 ms-latency music generation in
the Euterpe toolbox. Bridged to TypeScript (`@euterpe/realtime-gen`) via napi-rs,
mirroring the `@kalika/cas-engine` workspace.

## Crates

| Crate         | Type            | Responsibility |
|---------------|-----------------|----------------|
| `mrt2-core`   | rlib (pure)     | Frame/token **budget** math, on-device **feasibility** (numerically identical to `@euterpe/genesis` `frontier`), **tier** selection (2.4B/230M), the live **conditioning** model (MIDI/text/audio-ref + sliding-window context), and the SpectroStream **codec** descriptor. No I/O, no model. |
| `mrt2-engine` | rlib            | Real `.safetensors` header parsing + **bundle discovery/validation**, feasibility-gated **load**, and the **fail-closed inference seam**. |
| `mrt2-native` | cdylib + rlib   | napi-rs bridge exposing the above to Node. |

## Fail-closed by design

`mrt2-engine` does every weight-dependent thing it can without the model's
forward graph — it validates a downloaded bundle, counts real parameters, and
runs the device feasibility gate — then **reports `not_configured` honestly**
instead of fabricating audio. No code path ever returns invented PCM.

### Enabling generation is NOT "download weights → go" — it bridges the official engine

MRT2's published runtime is a **C++/MLX inference engine** (models compiled to an
`.mlxfn` weights+graph container) and the **`magenta-rt` Python package**
(`pip install magenta-rt`); weights live at
[`google/magenta-realtime-2`](https://huggingface.co/google/magenta-realtime-2),
code at [`github.com/magenta/magenta-realtime`](https://github.com/magenta/magenta-realtime).
The model is a **dual transformer** — `Temporal_θ` (compresses frame history) +
`Depth_φ` (depth-wise RVQ token decode) — with causal sliding-window attention
over the **SpectroStream** codec (48 kHz stereo, 25 Hz / 40 ms frames, 12 RVQ
levels × 1024 vocab, 3 kbps), conditioned by quantized MusicCoCa embeddings
(style), a 128-channel 25 Hz pianoroll (notes) and OaF binary drums.

This crate therefore does **not** reimplement that forward pass (the layer dims
aren't even public). Enabling generation means **bridging the official engine**
(an MLX/`magenta-rt` sidecar over local IPC, or FFI to the C++ engine) and
flipping `INFERENCE_BACKEND_LINKED` — a deploy-time integration that requires the
engine + weights on an Apple Silicon host. Until then everything fails closed.

## Commands

```bash
# From libs/euterpe/realtime-engine (bypasses Nx per the worktree rules):
cargo test --workspace          # unit tests (budget, feasibility, codec, safetensors, discovery)
cargo check --workspace         # typecheck
cargo fmt --all --check         # lint

# Stage the native .node addon (fail-closed engine):
scripts/build-native.sh         # → dist/libs/euterpe/realtime-engine/native/<platform>-<arch>/mrt2_native.node
```

**This crate does discovery / feasibility / codec math — it does NOT generate.**
Generation runs in the **`magenta_rt` MLX sidecar** (`tools/euterpe-mrt2-sidecar/`,
wired via `OSHUN_MRT2_SIDECAR`); see that README for the runbook. The Rust napi
path stays fail-closed.

## Environment

The Rust discovery helper (this crate, via the napi `NativeOnDeviceModelRuntime`):

| Var | Meaning |
|-----|---------|
| `OSHUN_MRT2_RUNTIME` | `auto` \| `native` \| `off` (default `auto`) |
| `OSHUN_MRT2_WEIGHTS_DIR` | Path to an unpacked bundle for the Rust **discovery** helper (NOT where generation weights live) |
| `OSHUN_MRT2_TIER` | `230m` \| `2.4b` (also selects the sidecar model size) |
| `OSHUN_MRT2_ENGINE_PATH` | Explicit `.node` addon path for the discovery helper |

The **generation** sidecar (the real engine; `tools/euterpe-mrt2-sidecar/`):

| Var | Meaning |
|-----|---------|
| `OSHUN_MRT2_ENABLED` | `true` to enable the realtime route |
| `OSHUN_MRT2_SIDECAR` | Absolute path to `magenta_rt_sidecar.py` (selects the sidecar over the Rust helper) |
| `OSHUN_MRT2_PYTHON` | Interpreter that has `magenta_rt` (default `python3`) |

Generation weights live in the `magenta_rt` checkpoints dir (`mrt models download`),
not `OSHUN_MRT2_WEIGHTS_DIR`.