Cross-Lingual Interleaving for Speech Language Models

TL;DR

This work tackles the English-centric bias in evaluating and training speech language models by introducing a textless cross-lingual interleaving method that mixes speech tokens from multiple languages. It demonstrates that sentence-aligned interleaving across English and French encourages a shared representational subspace, enabling cross-lingual continuation and improved cross-language alignment, while still benefiting monolingual semantics under a matched token budget. The authors release a French–English TinyStories corpus (~42k hours) and bilingual spoken StoryCloze and TopicCloze benchmarks, showing positive transfer at 360M and 1B parameter scales and stronger hidden-state alignment. The approach is presented as a scalable, practical pathway to multilingual SLMs that can understand and converse across languages, with resources openly available for reproducibility and further research.

Abstract

Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English-centric due to scarce spoken evaluation benchmarks and training data, making cross-lingual learning difficult. We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN-FR training dataset, TinyStories (~42k hours), together with EN-FR spoken StoryCloze and TopicCloze benchmarks for cross-lingual semantic evaluation, both synthetically generated using GPT-4. On 360M and 1B SLMs under matched training-token budgets, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. Taken together, these results indicate that cross-lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility.