VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model

TL;DR

VITA-Audio tackles the latency bottleneck in real-time end-to-end speech systems by introducing a lightweight, cross-modal token prediction mechanism (MCTP) that can generate multiple audio tokens within a single forward pass, achieving zero delay for the first audio chunk. A four-stage training strategy and four inference modes enable fast, interleaved text-audio generation without heavily compromising speech quality, attaining a three- to fivefold speedup at 7B-scale LLMs and strong performance on ASR, TTS, and SQA benchmarks. The approach preserves LLM language capabilities by interleaving audio generation with text processing and demonstrates real-time conversational potential with open-source data. Overall, VITA-Audio sets a new standard for real-time speech-to-speech models by combining efficient cross-modal mapping with end-to-end training and open-release accessibility.

Abstract

With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.

Figures (5)

• Figure 1: (a) The audio sequence generated by the speech model exhibits a strong correlation with the corresponding text tokens. (b) With irrelevant text tokens being masked out, the model is still able to generate the correct audio, and the pronunciation remains contextually appropriate. However, if all text tokens are masked, the model outputs random audio. This suggests that the hidden states from the LLM include sufficient contextual information for generating the corresponding audio tokens. Consequently, the mapping from text hidden states to audio tokens is accomplished using relatively simple modules, without the need for the extensive semantic modeling typically required by LLMs.
• Figure 2: Architecture overview. (a) VITA-Audio is an end-to-end large speech model equipped with 10 light-weight Multiple Cross-modal Token Prediction (MCTP) modules that enable speech generation with extremely low latency. As shown in \ref{['fig:attn_weights']}, we observe that the hidden states of certain text tokens in the LLM backbone contain sufficient semantic information for generating the corresponding audio tokens, which means that it is unnecessary to attend to additional text tokens when generating audio. Thus, we propose to utilize a set of light-weight MCTP modules to model the mapping from LLM hidden states to the audio tokens. (b) The details of the MCTP modules. Our MCTP module has a light-weight architecture, which enables it to finish one forward pass within $0.0024$ seconds ( $11$% of the LLM backbone). The MCTP module is capable of generating 10 audio tokens from the LLM hidden states and the text embedding, and the generated audio tokens can be decoded by the audio decoder directly. The utilization of MCTP modules enables VITA-Audio to generate audio responses in one LLM forward pass, which achieves extremely fast generation speed.
• Figure 3: Training pipeline of VITA-Audio. The first stage (Audio-Text Alignment) enhances the LLM by extending its audio modeling capability through large-scale speech pre-training. The second stage (Single MCTP module Training) connects an MCTP module with the LLM to predict one subsequent token based on the input tokens and the LLM's hidden states. The third stage (Multiple MCTP Modules Training) increases the number of MCTP modules in the model to predict more tokens in each model forward. The last stage (Supervised Fine-tuning) provides the speech-to-speech capability to the model by optimizing it on the large-scale speech QA dataset.
• Figure 4: The four text-audio interleaved inference modes are illustrated as follows: 1) Turbo: As the fastest inference mode, it generates $1$ token by the main model and $10$ additional tokens via MCTP in each forward pass. To ensure that a valid audio chunk is decoded after the first forward pass, the first generated $11$ tokens are split into $1$ text token and $10$ audio tokens. Then, the Turbo mode iteratively generates $4$ text tokens and $10$ audio tokens in the following forward. 2) Boost: To enhance the quality of text tokens, Boost mode follows the text-audio cyclic pattern of Turbo mode, with the main model generating every text token and MCTP generating every audio token. 3) Balance: To keep a balanced text-audio ratio, i.e., $1:2$, the balance mode further changes the text-audio cyclic pattern of the Boost mode. Specifically, the balance mode sequentially generates $1$ text token from the main model, $4$ audio tokens ($2$ tag tokens mark the beginning and end of audios, and $2$ common tokens denote the audio content) from MCTP, $3$ text tokens from the main model, $8$ text tokens ($2$ tag tokens mark the beginning and end of audios, and $6$ common tokens denote the audio content) from MCTP, and then iteratively generates $4$ text tokens from the main model and $10$ audio tokens ($2$ tag tokens mark the beginning and end of audios, and $8$ common tokens denote the audio content) from MCTP. 4) Vanilla: As the slowest inference mode, Vanilla mode follows the text-audio cyclic pattern of Balance mode, with the main model generating every token.
• Figure 5: Token generation speed curves of four text-audio interleaved modes.