How we made a text-to-speech model respond in sub-50 ms
Nari Labs has significantly advanced real-time text-to-speech with their optimized Qwen3-TTS implementation, achieving sub-50ms time-to-first-audio at 10 requests per second on a single H100. This technical deep dive showcases novel scheduling and architectural optimizations, pushing the boundaries for low-latency AI inference. The open-sourced work addresses critical bottlenecks for responsive voice agents, sparking discussion on quality trade-offs and on-device deployment.
The Lowdown
Nari Labs has announced a breakthrough in Text-to-Speech (TTS) latency with their optimized implementation of the Qwen3-TTS 1.7B CustomVoice model. Their work focuses on achieving extremely low time-to-first-audio (TTFA) and high throughput, critical for real-time voice applications.
- The team achieved a remarkable 34ms p95 TTFA at 10 requests per second (RPS) on a single NVIDIA H100 SXM, maintaining real-time playback. This significantly outperforms other existing implementations, including vLLM-Omni and SGLang-Omni.
- Their system can produce approximately 630 characters per second, with an estimated cost of ~$2 per 1 million characters at full utilization, presenting a substantial cost-efficiency compared to commercial offerings.
- Nari Labs open-sourced both their optimized implementation and benchmarking tools, fostering transparency and further development in the community.
- Key optimizations included dynamically trimming leading silence, fine-tuning frame accumulation, and introducing a novel scheduler. This scheduler coordinates the three distinct modules of Qwen3-TTS (Talker, Code Predictor, Codec) on a shared surface, allowing for intelligent prioritization of tasks.
- The system exploits the regular structure of the Code Predictor using CUDA graphs and specialized kernels, and rebuilds the Codec around cached state for efficient incremental decoding.
- Additional serving optimizations involve capturing CUDA graphs for predefined batch sizes and minimizing CPU-GPU synchronization to reduce overhead.
This work represents a significant step forward in the performance and cost-efficiency of real-time TTS. Nari Labs indicates plans to expand their research to other multimodal inference challenges, aiming for real-time simulation of the world.
The Gossip
Quality Conundrums
Commenters expressed curiosity and concern regarding the potential trade-offs between ultra-low latency and audio quality. While acknowledging the impressive speed, some highlighted that for truly effective voice agents, factors like natural cadence, expression, and overall voice quality are paramount, suggesting there's a 'quality hard wall' where further latency gains might not be worth the degradation in output.
Latency vs. LLM Lag
A significant portion of the discussion centered on how this TTS optimization fits into the broader context of AI voice agents, particularly concerning the latency introduced by the preceding Large Language Model (LLM) inference. Users noted that while the TTS might be fast, the overall user experience is still gated by the LLM's response time. Some drew comparisons to OpenAI's models, which, despite fast *perceived* responsiveness often employ filler words like 'hmm...' to mask underlying processing delays, raising questions about what truly constitutes 'real-time' and 'relevant' audio output.
On-Device Dreams
Many commenters voiced a strong desire for this kind of high-performance TTS to become viable on consumer-level hardware, specifically mobile devices, rather than requiring an H100 GPU. While acknowledging the current state of the art is impressive, the 'real win' for widespread adoption and truly ubiquitous voice agents was seen as enabling efficient, inexpensive on-device inference.
Practical Inquiries
Users also posed practical questions about the implementation and its demonstration. Requests for a video demonstration to hear the quality and experience the low latency firsthand were common. There was also interest in specific technical details, such as whether the benchmark numbers accounted for cold starts or warmed caches, to fully understand the real-world applicability.