Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-Centric Lessons To Improve Speech-Language Pretraining

Vishaal Udandarao, Zhiyun Lu, Xuankai Chang, Yongqiang Wang, Violet Z. Yao, Albin Madapally Jose, Fartash Faghri, Josh Gardner, Chung-Cheng Chiu

TL;DR

This work investigates data-centric improvements for speech-language pretraining to enhance Spoken Question-Answering (SQA). It conducts controlled ablations on three fronts—data processing with fine-grained interleaving, synthetic data augmentation (Krist and Quest), and deterministic modality sampling—demonstrating substantial SQA gains and robust transfer to text-only and post-training settings. The authors show that these interventions improve cross-modal alignment and expand domain coverage, enabling a 3.8B SpeechLM, SpeLangy, to outperform models up to 3x larger on SQA by about 10% on average. The findings underscore the central role of data curation in SpeechLM pretraining and offer concrete guidance for data-centric design, including risk analyses such as contamination checks. Overall, the paper presents a scalable, data-focused pathway to elevate speech-language models without proportionally increasing model size or compute.

Abstract

Spoken Question-Answering (SQA) is a core capability for useful and interactive artificial intelligence systems. Recently, several speech-language models (SpeechLMs) have been released with a specific focus on improving their SQA performance. However, a lack of controlled ablations of pretraining data processing and curation makes it challenging to understand what factors account for performance, despite substantial gains from similar studies in other data modalities. In this work, we address this gap by conducting a data-centric exploration for pretraining SpeechLMs. We focus on three research questions fundamental to speech-language pretraining data: (1) how to process raw web-crawled audio content for speech-text pretraining, (2) how to construct synthetic pretraining datasets to augment web-crawled data and (3) how to interleave (text, audio) segments into training sequences. We apply the insights from our controlled data-centric ablations to pretrain a 3.8B-parameter SpeechLM, called SpeLangy, that outperforms models that are up to 3x larger by 10.2% absolute performance. We hope our findings highlight the impact of effective data curation for speech-language pretraining and guide future data-centric exploration in SpeechLMs.

Data-Centric Lessons To Improve Speech-Language Pretraining

TL;DR

This work investigates data-centric improvements for speech-language pretraining to enhance Spoken Question-Answering (SQA). It conducts controlled ablations on three fronts—data processing with fine-grained interleaving, synthetic data augmentation (Krist and Quest), and deterministic modality sampling—demonstrating substantial SQA gains and robust transfer to text-only and post-training settings. The authors show that these interventions improve cross-modal alignment and expand domain coverage, enabling a 3.8B SpeechLM, SpeLangy, to outperform models up to 3x larger on SQA by about 10% on average. The findings underscore the central role of data curation in SpeechLM pretraining and offer concrete guidance for data-centric design, including risk analyses such as contamination checks. Overall, the paper presents a scalable, data-focused pathway to elevate speech-language models without proportionally increasing model size or compute.

Abstract

Spoken Question-Answering (SQA) is a core capability for useful and interactive artificial intelligence systems. Recently, several speech-language models (SpeechLMs) have been released with a specific focus on improving their SQA performance. However, a lack of controlled ablations of pretraining data processing and curation makes it challenging to understand what factors account for performance, despite substantial gains from similar studies in other data modalities. In this work, we address this gap by conducting a data-centric exploration for pretraining SpeechLMs. We focus on three research questions fundamental to speech-language pretraining data: (1) how to process raw web-crawled audio content for speech-text pretraining, (2) how to construct synthetic pretraining datasets to augment web-crawled data and (3) how to interleave (text, audio) segments into training sequences. We apply the insights from our controlled data-centric ablations to pretrain a 3.8B-parameter SpeechLM, called SpeLangy, that outperforms models that are up to 3x larger by 10.2% absolute performance. We hope our findings highlight the impact of effective data curation for speech-language pretraining and guide future data-centric exploration in SpeechLMs.
Paper Structure (61 sections, 6 equations, 16 figures, 15 tables, 1 algorithm)

This paper contains 61 sections, 6 equations, 16 figures, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controlled Data-Centric Experiments
  4. Evaluation Benchmarks
  5. Base Setup
  6. Processing pretraining data via fine-grained interleaving
  7. Constructing effective synthetic datasets
  8. Modality sampling schemes for interleaved training
  9. Our data-centric lessons transfer to understanding-only SpeechLMs
  10. Our data-centric lessons transfer after post-training
  11. Understanding Why Our Data Interventions Help
  12. Improved alignment between modality distributions
  13. Synthetic data improves domain coverage
  14. Analysing train-test contamination
  15. SpeLangy: Bringing it all together
  16. ...and 46 more sections

Figures (16)

  • Figure 1: (Left) We highlight the three data-centric questions we study in this work (\ref{['designspace']}), (Right) Distilling our insights from controlled data-centric experiments yields a strong $3.8$B-parameter SpeechLM, SpeLangy (\ref{['bringingtogether']}).
  • Figure 2: Our experimental conditions for speech-text pretraining data. (Top) We study two interleaving strategies: coarse (long chunks) and fine (short chunks) (\ref{['short-vs-long-segments']}). (Middle) We construct two synthetic datasets---Krist and Quest---from filtered knowledge-rich web-crawled documents (\ref{['synthetic-data-main']}). (Bottom) We study two modality-sampling schemes for interleaved training: deterministic and stochastic (\ref{['chunk-sampling']}).
  • Figure 3: Audio chunk length distribution (in seconds) for our two interleaving strategies.
  • Figure 4: Number of modality switches during interleaved training for our two sampling schemes.
  • Figure 5: Our methods reduce the distribution gap (reverse-KLD) between text and audio modalities.
  • ...and 11 more figures