From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM

TL;DR

Spire demonstrates that speech can be incorporated into an existing translation-focused LLM by framing discretized speech units as an additional translation language and applying a two-stage CPT then IT training regime. Using only $42.5K$ hours of speech and a DSU vocabulary expansion, Spire preserves Tower’s strong text MT performance while gaining ASR and ST capabilities, achieving competitive results against large, data-intensive baselines. The work provides a practical, reproducible recipe for modality expansion in multilingual LLMs and highlights the complementary roles of CPT and IT in achieving robust cross-modal translation tasks. This approach offers a data-efficient path to extending LLMs with speech processing for multilingual applications, with public code and models to foster adoption and further research.

Abstract

We introduce Spire, a speech-augmented language model (LM) capable of both translating and transcribing speech input from English into 10 other languages as well as translating text input in both language directions. Spire integrates the speech modality into an existing multilingual LM via speech discretization and continued pre-training using only 42.5K hours of speech. In particular, we adopt the pretraining framework of multilingual LMs and treat discretized speech input as an additional translation language. This approach not only equips the model with speech capabilities, but also preserves its strong text-based performance. We achieve this using significantly less data than existing speech LMs, demonstrating that discretized speech input integration as an additional language is feasible during LM adaptation. We make our code and models available to the community.

Figures (1)

• Figure 1: Illustration of the model training approach for SpireBase and SpireFull.