Spirit LM: Interleaved Spoken and Written Language Model

TL;DR

Spirit LM addresses the challenge of unifying text and speech understanding and generation by interleaving text and speech tokens during continuous pretraining of a foundation model. It introduces a Base variant using HuBERT phonetic tokens and an Expressive variant augmented with pitch and style tokens to capture prosody, enabling cross-modal generation and few-shot learning across modalities (ASR, TTS, Speech Classification). The paper demonstrates that interleaving is critical for cross-modal alignment and that expressivity incurs a modest modeling cost, while also proposing STSP to evaluate sentiment preservation across modalities. It discusses safety, toxicity, and broader impacts, and provides a pathway towards scalable, expressive multimodal LLMs with practical considerations for deployment and future research.

Abstract

We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code.

Figures (6)

• Figure 1: a. The Spirit LM architecture. A language model trained with next token prediction; tokens are derived from speech or text with an encoder, and rendered back in their original modality with a decoder. Spirit LM models are trained on a mix of text-only sequences, speech-only sequences, and interleaved speech-text sequences. b. Speech-text interleaving scheme. Speech is encoded into tokens (pink) using clusterized speech units (Hubert, Pitch, or Style tokens), and text (blue) using BPE. We use special tokens [Text] to prefix text and [Speech] for speech tokens. During training, a change of modality is randomly triggered at word boundaries in aligned speech-text corpora. Speech tokens are deduplicated and interleaved with text tokens at the modality change boundary. c. Expressive Speech tokens. For Spirit LMExpressive, pitch tokens and style tokens are interleaved after deduplication.
• Figure 2: Spirit LMBase performance with regard to the number of shots presented to the model context for Intent Classification, ASR and TTS.
• Figure 3: Performance of Spirit LMBase on Topic-StoryCloze in speech and text with regard to the sampled amount of aligned speech+text data from 0% to 100% out of the 8.4B aligned tokens (1.4B text and 7B speech).
• Figure 4: Comparing Spirit LMBase to a randomly initialized model trained in the same way and to a model trained with no Interleaving data. (i.e. the model is only trained on sequences of raw speech or raw text data without any interleaved aligned data.)
• Figure 5: Toxicity Distribution Relative Distribution of added toxicity over the 13 demographic axes for T$\rightarrow$T and S$\rightarrow$S generations. The number of added toxicities are normalized by the number of occurrences in each demographic axis.