To guarantee sub-600 ms time-to-first-byte for short, streaming utterances, the TTS service now maintains a pre-roll warmer pool.
Overview#
- Warm pool size: 5 buffers per voice (configurable in
services/tts/src/app.ts). - Eligibility: applied automatically when
stream=trueand the text/SSML payload is ≤ 240 characters. - Warm preparation: executed asynchronously at boot and replenished in the background whenever the pool drops below two ready buffers.
- Response path: warmed buffers are piped through a
PassThroughstream so the first chunk flushes immediately while the remaining payload drains asynchronously. - Visibility:
GET /v1/tts/warmers/statusexposes real-time availability (available,in_flight,target_pool,first_byte_target_ms).
Behaviour#
- Startup:
setupStreamingWarmers()seeds each registered voice with warmed buffers. - Request handling: When a short streaming request arrives, the handler checks out a warm buffer (if available) and instantly streams the first 2 KB chunk before completing the rest of the payload.
- Replenishment: Warm buffers are replenished asynchronously via
scheduleWarmPreparation(), preserving headroom for subsequent requests.
Testing#
- Compliance is covered by Playwright spec
tests/tts.streaming-warmers.spec.tswhich asserts:- Warmers are available before streaming.
- Time-to-first-byte is < 600 ms.
- Warm pool replenishes after use.
Operations#
- Use
/v1/tts/warmers/statusin dashboards to monitor warm availability. - Alert when
availabledrops to 0 for more than one minute—indicates insufficient warmers or sustained load. - Increase
STREAM_WARM_POOL_SIZEif sustained traffic routinely consumes the pool faster than replenishment.