Latent Speech-Text Transformer

TL;DR

The paper tackles the inefficiency and misalignment in auto-regressive speech–text models caused by long speech token sequences. It introduces Latent Speech-Text Transformer (LST), which patches speech tokens into latent units via a Patch Encoder and decodes back with a Patch Decoder, enabling efficient cross-modal modeling. It systematically evaluates multiple patching strategies (Static, Aligned, Mixed, Curriculum) and shows curriculum patching yields the strongest gains, achieving data- and compute-controlled improvements on S→S and T→T benchmarks, with notable gains on HellaSwag and StoryCloze and scaling benefits up to 7B parameters. The work demonstrates that aligning speech and text at the latent patch level reduces the information density gap and improves scalability, offering a practical path toward efficient multimodal Spoken NLP and cross-modal transfer.

Abstract

Auto-regressive speech-text models are typically pre-trained on a large number of interleaved sequences of text tokens and raw speech encoded as speech tokens using vector quantization. These models have demonstrated state-of-the-art performance in speech-to-speech understanding and generation benchmarks, together with promising scaling laws, primarily enabled by the representational alignment between text and speech. Nevertheless, they suffer from shortcomings, partly owing to the disproportionately longer sequences of speech tokens in contrast to textual tokens. This results in a large compute imbalance between modalities during pre-training as well as during inference, and a potential hindrance to effectively aligning speech and text, ultimately translating to several orders of magnitude slower scaling laws. We introduce the Latent Speech-Text Transformer (LST), which makes pre-training speech-text models more data-efficient by dynamically and inexpensively aggregating speech tokens into latent speech patches. These patches serve as higher-level units that can either align with corresponding textual units to aid capability transfer or even encapsulate common speech sequences like silences to be more compute-efficient. We show that LST outperforms vanilla approaches on speech-to-speech as well as text-to-text benchmarks in both data- and compute-controlled settings, the former indicating more effective representational alignment and the latter indicating steeper scaling laws for speech-text models. On HellaSwag story completion, LST achieves 6.5% absolute gain in speech accuracy under compute-controlled training and 5.3% under data-controlled training, while also improving text performance. We will release our models, code, and the evaluation data to facilitate further research.

Figures (8)

• Figure 1: Comparison of LST and Baseline on HellaSwag story completion under two experimental setups, (a) compute-controlled: same number of training iterations and (b) data-controlled: same amount of training data.
• Figure 2: Latent Speech-Text Transformer (LST). The model encodes BPE text tokens and HuBERT speech tokens into a shared latent space. A Patch Encoder compresses local speech segments into patch representations, which are jointly processed with text tokens. A Patch Decoder predicts future speech tokens from latent representations, enabling alignment and transfer across modalities.
• Figure 3: Illustrations of alignment and patching methods.
• Figure 4: Visualization of word-level speech patch embeddings from alignment patching models on HellaSwag speech, grouped by different linguistic categories. Clusters of the same word are tight and well-separated from others.
• Figure 5: Scaling trends of baseline SpeechLLM and LST models at 1B and 7B parameter scales. Each dataset reports both S$\rightarrow$S and T$\rightarrow$T.