Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Gap Between Text and Speech Understanding in LLMs

Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, Zakaria Aldeneh

TL;DR

This work tackles the persistent text-speech understanding gap by formalizing forgetting and cross-modal misalignment as key causes and showing their link to broad-domain task performance. It introduces SALAD, a two-stage, data-efficient method that combines cross-modal distillation with active selection to align speech inputs with text-based LLM backbones while preserving core text capabilities. Applied to 3B and 7B models, SALAD achieves competitive results with an order of magnitude less speech data than prior end-to-end approaches and demonstrates that domain-matched distillation further boosts performance. The findings highlight that careful objective design and targeted data augmentation can close the text-speech gap in a reproducible, data-efficient manner, improving spoken language understanding without extensive synthetic or proprietary datasets.

Abstract

Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on this analysis, we introduce SALAD--Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation--which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with a strong open-weight model across broad-domain benchmarks in knowledge, language understanding, and reasoning, while training on over an order of magnitude less speech data from public corpora.

Closing the Gap Between Text and Speech Understanding in LLMs

TL;DR

This work tackles the persistent text-speech understanding gap by formalizing forgetting and cross-modal misalignment as key causes and showing their link to broad-domain task performance. It introduces SALAD, a two-stage, data-efficient method that combines cross-modal distillation with active selection to align speech inputs with text-based LLM backbones while preserving core text capabilities. Applied to 3B and 7B models, SALAD achieves competitive results with an order of magnitude less speech data than prior end-to-end approaches and demonstrates that domain-matched distillation further boosts performance. The findings highlight that careful objective design and targeted data augmentation can close the text-speech gap in a reproducible, data-efficient manner, improving spoken language understanding without extensive synthetic or proprietary datasets.

Abstract

Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on this analysis, we introduce SALAD--Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation--which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with a strong open-weight model across broad-domain benchmarks in knowledge, language understanding, and reasoning, while training on over an order of magnitude less speech data from public corpora.
Paper Structure (37 sections, 10 equations, 9 figures, 8 tables)

This paper contains 37 sections, 10 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Analyzing the Text-Speech Gap
  4. Objective
  5. Data
  6. Benchmarks
  7. Architecture
  8. Results
  9. How do cross-modal misalignment and forgetting relate to broad-domain performance?
  10. How does the training objective interact with cross-modal alignment and forgetting?
  11. Does a better data domain match between speech and text training alleviate the issues of NLL training?
  12. Does a better data domain match between speech and text training yield gains for cross-modal distillation?
  13. Takeaways.
  14. Closing the Text-Speech Gap
  15. Method
  16. ...and 22 more sections

Figures (9)

  • Figure 1: SALAD reduces the text-speech understanding gap while requiring over an order of magnitude less training data than competing speech-adapted LLMs. Left: The gap between the performance of the base LLM given text inputs and the performance of the speech-adapted LLM given speech inputs as a function of training dataset size. Right: The performance of the base LLM given text inputs and the performance of the speech-adapted LLM given speech inputs across several language understanding tasks.
  • Figure 2: Distribution of samples across $64$ automatically annotated domains (x-axis). Text corpora (FineWeb-Edu; penedo2024fineweb) cover a wide range of domains, whereas speech datasets (LibriHeavy; libriheavykang24 and Emilia; he2024emiliaextensivemultilingualdiverse) are concentrated in only a few. The ten least represented domains in speech are labeled in grey.
  • Figure 3: Relationship between speech performance and cross-modal misalignment (Equation \ref{['eq:misalign']}) (left); relation between text performance and forgetting (Equation \ref{['eq:forgetting']}) (right).
  • Figure 4: Impact of the training objective (controlled by $\alpha$ in Equation \ref{['eq:total_loss']}), the number of training tokens (shown on the $x$-axis), and the dataset choice on cross-modal misalignment (Equation \ref{['eq:misalign']}), forgetting (Equation \ref{['eq:forgetting']}), and average speech performance.
  • Figure 5: Impact of the training objective (controlled by $\alpha$ in Equation \ref{['eq:total_loss']}), the number of training tokens (shown on the $x$-axis), and the dataset choice on the average text performance.
  • ...and 4 more figures