Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SSR: Alignment-Aware Modality Connector for Speech Language Models

Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomasello, Xutai Ma

TL;DR

This work introduces SSR-Connector, an alignment-aware modality connector that segments and compresses speech to match text token granularity by leveraging speech-text alignments. A two-stage training pipeline—distillation to align speech representations with the LLM embeddings, followed by fine-tuning with next-token prediction—mitigates catastrophic forgetting while enhancing cross-modal performance. Empirically, SSR-Connector improves speech understanding and cross-modal reasoning (e.g., StoryCloze +10 accuracy, Speech-MMLU +20) and achieves better ASR performance, while preserving pre-trained text capabilities. The approach demonstrates the value of explicit alignment-informed fusion for end-to-end SpeechLMs and highlights trade-offs between speech specialization and text retention in Stage 2 fine-tuning.

Abstract

Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.

SSR: Alignment-Aware Modality Connector for Speech Language Models

TL;DR

This work introduces SSR-Connector, an alignment-aware modality connector that segments and compresses speech to match text token granularity by leveraging speech-text alignments. A two-stage training pipeline—distillation to align speech representations with the LLM embeddings, followed by fine-tuning with next-token prediction—mitigates catastrophic forgetting while enhancing cross-modal performance. Empirically, SSR-Connector improves speech understanding and cross-modal reasoning (e.g., StoryCloze +10 accuracy, Speech-MMLU +20) and achieves better ASR performance, while preserving pre-trained text capabilities. The approach demonstrates the value of explicit alignment-informed fusion for end-to-end SpeechLMs and highlights trade-offs between speech specialization and text retention in Stage 2 fine-tuning.

Abstract

Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.
Paper Structure (28 sections, 4 equations, 5 figures, 11 tables)

This paper contains 28 sections, 4 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Modality Fusion for Speech Language Models
  4. Methodology
  5. Alignment-Aware Speech Representation Connector
  6. Training Method
  7. Stage 1: Alignment-Aware Connector Distillation
  8. Datasets
  9. Model Setup
  10. Experimental Results
  11. Spoken Language Understanding Performance
  12. Speech-MMLU and Prompt-based ASR Performance
  13. Beyond Semantics
  14. Stage 2: Speech Language Model Fine-tuning
  15. Mitigate Catastrophic Forgetting
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Comparison of different approaches for speech-text modality fusion. (a): compressor-based connector. (b): direct fusion with speech units. (c): our alignment-aware connector.
  • Figure 2: SSR-Connector compresses speech features using speech-text alignments. Features are transformed by a Decoder-only model and selected at boundary index of each segment.
  • Figure 3: Two-stage training pipeline for SpeechLM with our alignment-aware modality connector.
  • Figure 4: Comparison of different fine-tuning methods on StoryCloze ($S$) and MMLU benchmark.
  • Figure 5: t-SNE plots of text and speech representations after distillation.