LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM

TL;DR

LLMVoX introduces a decoupled, autoregressive streaming TTS module with 30M parameters that is agnostic to the base LLM, enabling real-time speech without altering the LLM's reasoning. By tokenizing speech into discrete neural audio tokens and using ByT5-derived phonetic embeddings within a lightweight decoder, it achieves low latency (≈475 ms end-to-end) and strong speech-text alignment (WER ≈3.70) while preserving LLM capabilities. The multi-queue streaming inference supports infinite-length dialogues and seamless integration with vision-language models to form omni-modal systems, further demonstrated in Arabic generalization (CER ≈8%) and VSQA tasks with reduced latency. Overall, LLMVoX provides a scalable, plug-and-play pathway to high-quality speech in diverse LLM-based applications without expensive fine-tuning.

Abstract

Recent advancements in speech-to-speech dialogue systems leverage LLMs for multimodal interactions, yet they remain hindered by fine-tuning requirements, high computational overhead, and text-speech misalignment. Existing speech-enabled LLMs often degrade conversational quality by modifying the LLM, thereby compromising its linguistic capabilities. In contrast, we propose LLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS system that generates high-quality speech with low latency, while fully preserving the capabilities of the base LLM. Our approach achieves a significantly lower Word Error Rate compared to speech-enabled LLMs, while operating at comparable latency and UTMOS score. By decoupling speech synthesis from LLM processing via a multi-queue token streaming system, LLMVoX supports seamless, infinite-length dialogues. Its plug-and-play design also facilitates extension to various tasks with different backbones. Furthermore, LLMVoX generalizes to new languages with only dataset adaptation, attaining a low Character Error Rate on an Arabic speech task. Additionally, we have integrated LLMVoX with a Vision-Language Model to create an omni-model with speech, text, and vision capabilities, without requiring additional multimodal training. Our code base and project page is available at https://mbzuai-oryx.github.io/LLMVoX .

Figures (6)

• Figure 1: Speech quality (WER) vs latency (milliseconds) comparison of recent speech-enabled LLMs. Our LLMVoX is LLM-agnostic streaming TTS that generates high-quality speech (lower WER) comparable to XTTS casanova2024xtts while operating $~10$× faster. In the plot, $\triangle$ represents LLM-dependent methods, and $\bigstar$ denotes LLM-agnostic methods. The size of each symbol is proportional to the GPT score, indicating overall response quality. All methods are evaluated under similar settings and use similarly sized base LLMs.
• Figure 2: Overview of the proposed architecture. Text from the LLM is tokenized via a ByT5-based Grapheme-to-Phoneme(G2P) model, producing byte-level phoneme embeddings (teal). These are concatenated with the previous speech token’s feature vector (blue), L2-normalized, and fed into a decoder-only Transformer to generate the next token. A neural codec (WavTokenizer) decoder (orange) reconstructs speech every n speech tokens predicted.
• Figure 3: Overview of our streaming inference pipeline. Two replica TTS modules process text in parallel from two queues and place them into two producer queues.
• Figure 4: Human evaluation: Comparing with Freeze-Omni on Answer Relevance and Speech Quality.
• Figure 5: Breakdown of average end-to-end latency (in milliseconds) at a chunk size of 40 for a single query.