Eve library

Streaming Pre-Warmers for Lilith TTS

Guide

Revision 1

On this page

To guarantee sub-600 ms time-to-first-byte for short, streaming utterances, the TTS service now maintains a pre-roll warmer pool.

Overview#

  • Warm pool size: 5 buffers per voice (configurable in services/tts/src/app.ts).
  • Eligibility: applied automatically when stream=true and the text/SSML payload is ≤ 240 characters.
  • Warm preparation: executed asynchronously at boot and replenished in the background whenever the pool drops below two ready buffers.
  • Response path: warmed buffers are piped through a PassThrough stream so the first chunk flushes immediately while the remaining payload drains asynchronously.
  • Visibility: GET /v1/tts/warmers/status exposes real-time availability (available, in_flight, target_pool, first_byte_target_ms).

Behaviour#

  1. Startup: setupStreamingWarmers() seeds each registered voice with warmed buffers.
  2. Request handling: When a short streaming request arrives, the handler checks out a warm buffer (if available) and instantly streams the first 2 KB chunk before completing the rest of the payload.
  3. Replenishment: Warm buffers are replenished asynchronously via scheduleWarmPreparation(), preserving headroom for subsequent requests.

Testing#

  • Compliance is covered by Playwright spec tests/tts.streaming-warmers.spec.ts which asserts:
    • Warmers are available before streaming.
    • Time-to-first-byte is < 600 ms.
    • Warm pool replenishes after use.

Operations#

  • Use /v1/tts/warmers/status in dashboards to monitor warm availability.
  • Alert when available drops to 0 for more than one minute—indicates insufficient warmers or sustained load.
  • Increase STREAM_WARM_POOL_SIZE if sustained traffic routinely consumes the pool faster than replenishment.