Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners

TL;DR

Flow-Omni tackles robustness gaps in discrete speech-token multimodal models by adopting continuous mel-spectrogram tokens and a flow-matching generation mechanism. It integrates a Whisper-based speech encoder, a pretrained LLM backbone, and a flow-based speech output module to predict mel frames conditioned on speech prompts, enabling real-time, low-latency speech-to-speech interaction. The model is trained with a joint objective combining $L_{ ext{LM}}$ and $L_{ ext{CFM}}$ and uses a two-stage training regime for modality alignment followed by full fine-tuning, validated on Mandarin datasets where it outperforms discrete-token baselines in robustness and speech quality. This codec-free, continuous-token approach reduces representation losses from audio codecs, enhances robustness across domains, and supports end-to-end multimodal communication in practical dialogue systems.

Abstract

Recent advances in GPT-4o like multi-modality models have demonstrated remarkable progress for direct speech-to-speech conversation, with real-time speech interaction experience and strong speech understanding ability. However, current research focuses on discrete speech tokens to align with discrete text tokens for language modelling, which depends on an audio codec with residual connections or independent group tokens, such a codec usually leverages large scale and diverse datasets training to ensure that the discrete speech codes have good representation for varied domain, noise, style data reconstruction as well as a well-designed codec quantizer and encoder-decoder architecture for discrete token language modelling. This paper introduces Flow-Omni, a continuous speech token based GPT-4o like model, capable of real-time speech interaction and low streaming latency. Specifically, first, instead of cross-entropy loss only, we combine flow matching loss with a pretrained autoregressive LLM and a small MLP network to predict the probability distribution of the continuous-valued speech tokens from speech prompt. second, we incorporated the continuous speech tokens to Flow-Omni multi-modality training, thereby achieving robust speech-to-speech performance with discrete text tokens and continuous speech tokens together. Experiments demonstrate that, compared to discrete text and speech multi-modality training and its variants, the continuous speech tokens mitigate robustness issues by avoiding the inherent flaws of discrete speech code's representation loss for LLM.

Figures (3)

• Figure 1: The overview of Flow-Omni. Unlike discrete tokens based language modeling approaches, Flow-Omni generates the continuous mel-spectrogram conditioned on speech prompt autoregressively, using a single-stage decoder-only model its backbone, a whisper and adapter based model as speech understanding module, a light output adapter including flow matching MLP layers as speech generation module.
• Figure 2: The architecture of speech input and output module. The feature of the input audio is first extracted by Whisper encoder and passed through the adapter to match the dimension of LLM. The LLM hidden is preprocessed by several transformer blocks and then a simple MLP based flow matching predictor is used to predict the mel vector with time step and LLM hidden as condition.
• Figure 3: Parallel sequence of different tasks. When the input to the model is in either audio or text format, the other modality is filled with pad tokens. The text response follows the same format as in LLMs, and it is generated in parallel with the speech response, which is used to guide the synthesis of speech.