Scaling Properties of Speech Language Models

TL;DR

This work investigates how Speech Language Models (SLMs) trained on raw audio scale in relation to compute, parameters, and data, by applying established neural LM scaling laws. It shows a strong link between pretraining loss and downstream syntax/semantics, but reveals that SLMs require substantially more compute to achieve comparable linguistic performance to text-based LLMs, scaling up to three orders of magnitude more slowly. The authors introduce sTinyStories, a synthetic spoken dataset that improves semantic understanding, and find that coarser unigram tokenization does not help downstream performance. Overall, the study provides a quantitative framework for allocating compute to reach targeted linguistic capabilities in SLMs, while highlighting practical challenges and potential hybrid approaches with text-based models for real-world applications.

Abstract

Speech Language Models (SLMs) aim to learn language from raw audio, without textual resources. Despite significant advances, our current models exhibit weak syntax and semantic abilities. However, if the scaling properties of neural language models hold for the speech modality, these abilities will improve as the amount of compute used for training increases. In this paper, we use models of this scaling behavior to estimate the scale at which our current methods will yield a SLM with the English proficiency of text-based Large Language Models (LLMs). We establish a strong correlation between pre-training loss and downstream syntactic and semantic performance in SLMs and LLMs, which results in predictable scaling of linguistic performance. We show that the linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs. Additionally, we study the benefits of synthetic data designed to boost semantic understanding and the effects of coarser speech tokenization.

Figures (5)

• Figure 1: Speech Language Models test loss curves for all our single-epoch runs. Axes are in logarithmic scale. The envelope of minimal loss per FLOP (black dots) follows a power law (dashed line).
• Figure 2: Downstream linguistic performance scaling with compute for LLMs and SLMs. Axes are in logarithmic scale. Syntactic (BLIMP) and semantic (Topic Cloze and Story Cloze) metrics follow a power law before starting to saturate. Linguistic performance scales up to three orders of magnitude more slowly in SLMs relative to LLMs.
• Figure 3: Gains from synthetic data on downstream semantic performance of SLMs. Pre-training on sTinyStories yields consistent improvements on semantic understanding relative to pre-training on audiobooks (LibriSpeech plus LibriLight). Performance gains hold for mismatched train and test speakers.
• Figure 4: Correlation between downstream linguistic performance and test loss for LLMs and SLMs. Syntactic (BLIMP) and semantic (Topic Cloze and Story Cloze) metrics are strongly linearly correlated with the upstream test loss before saturation.
• Figure 5: Comparison of the scaling behavior of SLMs trained on raw speech tokens and unigram compressed tokens. Axes are in logarithmic scale. The upstream loss of SLMs trained on unigram tokens scales better with compute, but downstream performance scales worse. Notably, the Story Cloze metric for SLMs trained on unigram tokens does not seem to improve with increased compute.