Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

Shreyas Gopal, Donghang Wu, Ashutosh Anshul, Yeo Yue Heng, Yizhou Peng, Haoyang Li, Hexin Liu, Eng Siong Chng

TL;DR

This work synthesizes Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions, and introduces language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector.

Abstract

Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.

Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

TL;DR

This work synthesizes Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions, and introduces language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector.

Abstract

Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.
Paper Structure (20 sections, 4 equations, 1 figure, 3 tables)

This paper contains 20 sections, 4 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model Architecture
  4. Language-Aware Distillation
  5. Query Bank and Gating Network
  6. Scheduled Teacher Forcing
  7. Training Objective
  8. Experiments
  9. Training Data
  10. Evaluation Data
  11. Open-Ended Evaluation Dataset
  12. Close-Ended Evaluation Dataset - Audio--MLQA
  13. Evaluation Metric
  14. Experimental Setup
  15. Results
  16. ...and 5 more sections

Figures (1)

  • Figure 1: End-to-end model architecture. Trainable components are colored orange.