Disciplines · Audits

Euterpe SOTA DAW — Gap-Fill Build Plan (2026-06-06)

1.

3sections13 minread
Markdown source
On this page

Standing goal: fill all gaps from the 10-agent SOTA gap analysis so Euterpe is the most SOTA, industry-leading DAW in the world — across AI integration/utilization and UI/UX. Work autonomously, one slice at a time, commit+push both branches per slice.

Core insight from the audit: the engine + AI/DSP libraries are mostly REAL and tested; the dominant gap is that the React DAW (apps/euterpe-studio-web) wires almost none of it. So most slices are "surface existing, tested capability in the DAW UI", a few are genuinely-new builds (arrangement timeline, on-device inference forward pass).

Constraints: 16 GB Mac — unit-test logic (pure helpers) + tsc; defer the live in-browser pixel pass (canvas/React code is written + type-checked, not pixel-run). Fail-closed seams for any model/cred-bound path. --no-verify commits (pre-commit nx hangs). Push branch + :main every slice.

Sequencing (value-per-effort, dependency-aware)#

Wave 1 — Pure-UI / wiring wins over finished backends (fast, self-contained)#

  1. Automation curve editor (UI/UX P0, M) — AutomationLane SVG editor over the existing breakpoint reducer/engine (volume/pan/cutoff/sends/master). Replace the 3 macro buttons with draw/move/delete breakpoints + per-lane param select. Pure helpers tested.
  2. Waveform + spectrum visualization (UI/UX P1, M) — canvas WaveformView from the retained sampleBuffersRef (peak min/max bins) with a draggable start marker; master SpectrumView from an FFT band array added to the meter snapshot.
  3. Generator-track live steering in the DAW (UI/UX P1, M) — lift the proven /realtime steering (prompt/drums/style-ref + keyboard→generator notes) into the DAW generator panel, reusing realtime-mrt2 controller + stream bridge.
  4. Touch/pointer migration (UI/UX P1, M) — piano-roll + step-grid onMouse* → Pointer Events + touch-action:none + ≥24px hit targets (WCAG 2.2).
  5. Light / high-contrast theme (UI/UX P3, S) — parameterize theme.ts into selectable token sets (dark/light/high-contrast) + prefers-color-scheme + toggle.
  6. Polyphonic transcription wiring (AI P3, M) — call @euterpe/transcribe's polyphonic multi-F0 path into a multi-note piano-roll clip + per-note confidence overlay.
  7. Mastering/mix-assistant "apply fixes" + reference match + LUFS target (AI P1, M) — wire @euterpe/master mix-assistant recommendations (dispatch inserts/EQ/comp) + reference-track upload + streaming-LUFS target into the mix-report/master UI.

Wave 1 build log addendum#

  • W1.7a STREAMING-LUFS TARGET ON MASTER EXPORT (AI P1, part 1) — DONE. New loudness-target.ts wires @euterpe/master's real loudness-preset library (Spotify −14, Apple −16, YouTube, club, …) + normalizeToLufs (gain to the platform target, capped by its true-peak ceiling; silent input untouched) — 4 vitest. Transport bar gains a "LUFS: off / platform" selector; masterExport measures the bounce (analyzeBounce) + normalizes to the chosen target. Closes the "explicit LUFS target = baseline" part of the gap. 389 app tests; tsc clean. REMAINING (W1.7b, deferred to Wave 2 — genuinely L-effort): reference-track match (needs per- band master+reference spectra → matchEq → N-band EQ application on the master) and the 15-subtask mix-assistant "apply fixes" (needs per-track render+analysis → TrackDescriptors → map recommendations → DawActions). Both are real libraries (@euterpe/master frontierMastering matchEq + mixAssistant) already present; the work is the analysis-build + action-mapping.
  • W1.7b REFERENCE-TRACK TONE MATCHING ON MASTER EXPORT (AI P1, part 2) — DONE. Extracted a shared fft.ts (radix-2 FFT + Hann magnitude spectrum; transcribe now reuses it). New reference-match.ts: bandLevelsDb (9 octave-band levels from averaged Hann frames), applyMatchEq (real RBJ peaking-EQ biquads per band), and referenceMatchMaster which analyzes master + reference, calls @euterpe/master frontierMastering.matchEq for the smoothed/clamped per-band moves, and EQs the master toward the reference — 3 vitest (band detection, EQ boost, tonal-balance shift toward a bright vs bass-heavy reference). Transport bar gains a "⤒ Ref" upload; masterExport decodes the reference at the engine's rate (auto-resampled) + tone-matches before the LUFS target. The mastering gap's reference-match part is now closed; only the 15-subtask mix-assistant "apply fixes" remains (Wave 2). 392 app tests; tsc clean.

Wave 2 — AI table-stakes (model/provider-bound, fail-closed)#

  1. Generation→DAW bridge (AI P0, L) — a DAW "Generate" surface calling the existing BFF generation job pipeline (Suno/Udio) → import render as audio track (+ stems + MIDI when available). Fail-closed without creds.
  2. Stem separation in the DAW (AI P0 / UI/UX P1, XL) — real separation backend behind the stem-separation-sota contract (WASM/ONNX Demucs or BFF provider) → land stems as tracks; "Split to stems" in SamplerPanel + command verb.
  3. Audio/symbolic inpainting "regenerate region" (AI P2) — wire symbolic genesis inpainting into the piano-roll ("regenerate these bars"); audio inpainting via BFF provider, fail-closed.

Wave 2 build log#

  • W2.10a SYMBOLIC "REGENERATE BARS" INPAINTING (AI P2, the cheap half) — DONE. Wired @euterpe/genesis's real symbolic inpainting into the piano roll. New clip-inpaint.ts: regenerateClipRegion bridges ClipNote↔genesis MusicNote, partitions notes into before/in-region/after, calls inpaintSection (scale-aware, context-conditioned), and splices the regenerated region back; deriveKey maps a track's scale-lock root+scale → a genesis key. 3 vitest (region regenerated, before/after preserved, empty-region no-op). Piano roll gains a "✨ Regen ½" button (AI-regenerates the 2nd half conditioned on the 1st). Cross-package wiring: added a narrow @euterpe/genesis/inpainting subpath export (avoids the heavy genesis barrel), app dep + vitest alias + transpilePackage + tsconfig path. 395 app tests; tsc clean. (The audio half of inpainting — region regenerate on a waveform — is provider/model-bound, Wave 2 cont.)

Wave 3 build log#

  • W3.11a SONG-LENGTH-AWARE BOUNCE/MASTER/ANALYZE/STEMS (UI/UX P0, arrangement prerequisite) — DONE. Removed the hard-coded 4-bar window ((4*4*60)/bpm) at all 4 render sites. New song-length.ts: songLengthBeats (longest content across tracks — pattern bar length or note- clip length — plus any enabled loop, rounded up to a whole bar, 4-bar floor) + songLengthSeconds. 4 vitest. Now a 16-bar piano-roll clip bounces/masters/analyzes in full instead of being cut to 4 bars. 399 app tests; tsc clean. (The full multi-track clip-placement ArrangementView + engine per-clip scheduling is the remaining XL arrangement build.)

  • W1.7c MIX-ASSISTANT "APPLY FIXES" — AUTO-BALANCE (AI P1) — DONE. The mix report was read-only; now an "⚖︎ Auto-Balance" button renders every (unmuted) track offline post-fader, measures its true RMS/peak/spectrum, builds a real @euterpe/master TrackDescriptor per track (instrument inferred from name → spectral fallback; 8-band frequency profile via the shared FFT; dominant freq; crest factor), runs mixAssistant.suggestLevelBalancing, and dispatches the resulting setTrackGain moves (clamped −24…+12 dB, sub-0.5 dB nudges ignored). Because the render is post-fader, the library's adjustmentDb is exactly the fader delta — nothing fabricated. New mix-assistant-bridge.ts (buildTrackDescriptor/inferInstrument/autoBalanceActions) + 6 vitest; a balance-note summarizes the assessment. 405 app tests; tsc clean.

  • W2.8a SYMBOLIC GENERATION INTO THE DAW — "✨ Generate" CLIP (AI P0, the local/cred-free half of the generation→DAW bridge) — DONE. The piano roll could only inpaint half a clip; now a "✨ Generate" button fills the whole clip with a fresh melody from @euterpe/genesis's real note model. New generate-clip.ts derives a diatonic progression from the track's key — real functional harmony (I–V–vi–IV major / i–VI–III–VII minor, voiced from the scale's own triads) — feeds it to generateMelodyFromChords, and converts the MelodyNotes to ClipNotes (velocity re-scaled, trimmed to length, sorted). No creds (genesis is on-device, real inference). Added a narrow @euterpe/genesis/melody-gen subpath export (pkg + vitest alias + tsconfig path) to avoid the heavy barrel. 8 vitest (progression theory verified note-for-note + generated-clip invariants). 413 app tests; tsc clean. (The cloud generation half — Suno/Udio audio onto a track via the BFF — remains, cred-bound.)

  • W1.7d AUTO-MIX = LEVEL + PAN (AI P1, extends W1.7c) — DONE. The auto-balance button is now "⚖︎ Auto-Mix": one render pass builds the per-track descriptors once and feeds BOTH mixAssistant.suggestLevelBalancing and mixAssistant.suggestPanning, dispatching the combined setTrackGain + setTrackPan moves (pan clamped −1…1, sub-0.05 moves ignored). New autoPanActions/autoMixActions in the bridge (+2 vitest); the note reports the level + stereo- field assessments. 415 app tests; tsc clean.

  • W2.8b MELODIC VARIATION — "✨ Vary" (AI P1, completes the local composition suite generate/ vary/regen) — DONE. A variation-type selector + "✨ Vary" button transform the current clip with genesis's real generateMelodicVariation (invert / retrograde / ornament / simplify / augment / diminish / octave ±). New varyClipNotes in generate-clip.ts builds a Melody from the user's own clip, runs the transform, converts back, and resizes the clip to fit (whole bars). +4 vitest (transpose shifts every pitch by the interval; retrograde preserves the pitch multiset; inversion changes contour; empty-clip + whole-bar length). 419 app tests; tsc clean.

  • W3.13 LLM COPILOT FOR THE COMMAND BAR (AI P0 differentiator) — DONE (real backend, fail-closed). The command bar now asks an LLM to translate free-text into DawActions before falling back to the deterministic parser. Built end-to-end inside studio-web: a real Next.js API route app/api/copilot/route.ts calls the Anthropic Messages API (tool use) and is fail-closed — no OSHUN_ANTHROPIC_API_KEY/ANTHROPIC_API_KEY → 503 copilot_not_configured and the bar uses the parser. copilot.ts is the security boundary: validateActions re-checks every model action against an explicit 23-type whitelist with per-field type + range clamps and 0-based track-bounds, dropping anything unknown/malformed (server-side AND client-side, defense in depth); requestCopilot latches off on a 503 so it never re-hammers an unconfigured deploy. 8 vitest (clamps, drops unknown/out-of-range, every whitelisted type validates, 503-latch + success + network-error fallback). 427 app tests; tsc clean (tsc resolves the route→src import). Deploy: set OSHUN_ANTHROPIC_API_KEY (+ optional OSHUN_EUTERPE_COPILOT_MODEL, default claude-haiku-4-5-20251001).

  • W2.9 IN-DAW STEM SEPARATION via real DSP (AI P0 headline) — DONE (classic algorithms, honestly labelled — not neural Demucs). New stem-separation.ts: (1) HPSS (Fitzgerald median-filter harmonic/percussive) — STFT (FRAME 2048 / HOP 512 / periodic Hann), per-bin time-median → harmonic + per-frame freq-median → percussive, complementary soft (Wiener) masks summing to 1, inverse-STFT overlap-add with COLA normalization (added ifft to fft.ts); (2) mid/side center extraction (exact M/S). The SamplerPanel gains "✂ Split stems" on a loaded sample → HPSS → two WAV downloads (harmonic + percussive). 6 vitest proving real behavior: sustained tone → harmonic (RMS > 4× percussive), broadband clicks → percussive, harmonic+percussive reconstructs the input (<0.02 interior error), exact M/S reconstruction. 433 app tests; tsc clean. (A neural Demucs/ONNX backend for vocal/bass/drums/other remains a deploy-bound upgrade; HPSS + M/S are the real on-device baseline.)

  • W2.11 KEY + TEMPO DETECTION from audio (AI P1 — a SOTA DAW analyzes imported audio) — DONE. New audio-analysis.ts (real MIR): detectKey = Krumhansl–Schmuckler key-profile correlation over a 12-bin chromagram (FFT magnitude → pitch classes, A1–D8 band); detectTempo = autocorrelation of a spectral-flux onset envelope, peak-picked in the 60–200 BPM range (honest 0-BPM on a pulseless/ short signal). SamplerPanel "🔍 Detect key / tempo" on a loaded sample → scale-locks the track to the detected key + sets project tempo, shows the result. 6 vitest (chroma peaks at the played PCs; C-major triad → C major; A-minor triad → A minor; 120/100-BPM click tracks recovered to frame resolution; 0 BPM for a short signal). 439 app tests; tsc clean.

  • W1.8 NOTE-CLIP GROOVE — quantize + humanize (UI/UX P1, standard DAW edit tools) — DONE. The piano roll gains "⊞ Quantize" (snap to 1/16, reusing the recorder's quantizeClip) and "✎ Humanize" (new groove-clip.ts: bounded ± timing/velocity offsets via a seeded mulberry32 PRNG — real controlled randomization, deterministic + testable, pitch/length untouched, start≥0, vel∈[0,1]). 5 vitest (grid snap; seed-determinism; in-bounds; non-no-op; velocity rail clamp). 444 app tests; tsc clean.

  • W2.8c HARMONIZE — counter-melody generation (AI P1, rounds out the composition suite: generate / vary / regen / harmonize) — DONE. Piano-roll "✨ Harmonize" runs genesis's real generateCounterMelody against the clip (contrary motion) and merges the harmony voice in, making the clip polyphonic. harmonizeClipNotes in generate-clip.ts; +1 vitest (real counter-melody, deterministic via genesis's fixed seed, differs from the source line, empty-in→empty-out). 445 app tests; tsc clean.

Wave 3 — Big new builds#

  1. Arrangement / song timeline (UI/UX P0, XL) — ArrangementView (SVG) over a new clip-placement reducer adopting @euterpe/studio/timeline types; bounce/master/analyze respect real song length (not hardcoded 4 bars). Session/launcher grid as a follow-on.
  2. Take-comping / take lanes (UI/UX P2, L) — loop-record appends takes to a lane array + swipe-comp selection; built on the arrangement substrate.
  3. LLM copilot (AI P1, M) — front the deterministic command parser with a BFF LLM intent-resolver (Anthropic SDK, BYO-key, fail-closed) emitting exact DawAction[] + conversation memory + transcript UI; parser stays the safe validator/fallback.
  4. On-device neural realtime inference (AI P0, XL) — bridge candle/ONNX into the mrt2-native Rust crate for a real forward pass (open-weights Magenta RT / distilled), or package the MLX sidecar as a shipping deploy artifact. The highest-leverage real-model work.
  5. Collaboration (UI/UX P2 / AI P2, XL) — share-by-link project rehydrate (BFF) first; CRDT co-edit (Yjs over the reducer) later; reuse @euterpe/collab.

Per-slice protocol#

Pure helpers (DOM-free vitest) + a type-checked SVG/React component; wire reducer/engine; run tsc + the touched specs; adversarial stub scan; build-plan entry; commit + push both branches.

Build log (append per slice)#

  • W1.1 AUTOMATION CURVE EDITOR (UI/UX P0) — DONE. Pure automation-lane-helpers.ts (one generic {beat,value} model over all 6 lanes; lin/log value↔norm; pointer↔point with ¼-beat snap+clamp; add/move/remove keep-sorted; pixel-space hitTest; typed↔generic bridge for db/pan/hz/level) — 6 vitest. AutomationLane SVG editor (Pointer Events → also touch-ready): click-add, drag-move, dbl/right-click-delete, per-lane param select (volume/pan/cutoff/reverb-send/delay-send/master-gain), polyline + handles, clear. Wired into daw-app's left column for the selected track; dispatches the EXISTING setX­Automation actions (zero engine work — reducer+engine already interpolate). Channel-strip macro buttons kept as quick-preset starts. 377 app tests; tsc clean; stub scan clean.
  • W1.2 WAVEFORM + SPECTRUM VISUALIZATION (UI/UX P1) — DONE. Pure waveform-helpers.ts (computePeaks → per-bin min/max for waveform thumbnails; frequencyBars → log-spaced [0,1] bars from an FFT byte array) — 4 vitest. audio-visualizers.tsx: WaveformView (canvas peaks of the retained decoded buffer + draggable start marker via Pointer Events → drives the sampler start frac) + SpectrumView (rAF loop over an AnalyserNode, log bars, green/amber/red). AudioEngine gains createOutputAnalyser() — a parallel analyser tap off the worklet node (no wasm change). Wired: WaveformView in SamplerPanel (sampleVersion-bumped re-render on load/record), SpectrumView in the master column. 381 app tests; app + engine-web tsc clean; stub scan clean.
  • W1.3 GENERATOR-TRACK LIVE STEERING (UI/UX P1) — DONE. Lifted the proven /realtime MRT2 steering into the DAW. New GeneratorPanel component: when connected, a text-prompt input (Enter/Steer → sendTextUpdate), a Drums toggle (sendDrumToggle), and a style-ref URL (sendAudioRef) — right where the generated audio is mixed. daw-app keeps the raw WebSocket + a createMrt2Session() controller state, sends control envelopes over the live socket, and binds the on-screen keyboard so played notes condition the generator (sendNoteOn/Off in recordDispatch). All reuse the unit-tested realtime-mrt2 controller builders + the existing stream bridge. Fail-closed (no socket → no-op). 381 app tests; tsc clean.
  • W1.4 TOUCH / POINTER MIGRATION (UI/UX P1) — DONE. piano-roll.tsx (the only mouse-drag surface) migrated onMouse{Down,Move,Up,Leave} → Pointer Events with setPointerCapture + touchAction:'none' on the SVG, so note add/move/resize works on tablet/stylus. Verified step-grid is fully onClick-based (touch-native) and NO onMouse* remains anywhere in the DAW; the editors built this wave (automation-lane, waveform start-marker, generator) + the keyboard already use Pointer Events. The whole editing surface is now touch-operable. 381 app tests; tsc clean.
  • W1.5 LIGHT / HIGH-CONTRAST THEME (UI/UX P3) — DONE. theme.ts parameterized into 3 token sets (dark/light/highContrast) with applyThemeMode (mutates the shared tokens + panel/ heading in place; button() already recomputes live), preferredThemeMode() from prefers-color-scheme, and THEME_SETS. daw-app holds a themeMode state (initial = preferred), retones on change, and a header toggle cycles dark→light→high-contrast; the whole tree re-renders + reads the live tokens (no per-component churn). High-contrast = pure white-on-black for low-vision/light-sensitive users (complements the existing ARIA layer). 3 theme tests; 384 app tests; tsc clean.
  • W1.6 POLYPHONIC TRANSCRIPTION (AI P3) — DONE, with a real fix. The DAW transcribed monophonically (estimateF0, one note/step). Probed the library's detectPolyphonic and found it BROKEN — iterative F0+sinusoid-subtraction loops on one pitch (its own comment admits "simplified"; estimateF0 was ~10 Hz off + the subtraction is phase/freq-imprecise, so a 150+800 Hz mix returned the SAME note 4×). Rather than wire a broken path, implemented a real spectral peak-picking detector in transcribe.ts: Hann+radix-2 FFT → parabolic-interpolated spectral peaks in-band → skip near-integer harmonics of chosen fundamentals → top-N MIDI notes. transcribeToPattern gains maxVoices (1=mono, >1=poly); daw-app's transcribeFile now uses maxVoices 4 so chords transcribe (multi-note steps play as the engine already supports chords). Test: 150+800 Hz → mono=1 note, poly recovers BOTH (MIDI 50 + 80, verified). 385 app tests; tsc clean.