Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models

TL;DR

The paper systematically characterizes the modality gap between speech and text inputs in end-to-end Large Speech Language Models, revealing that deeper layers become directionally more aligned while magnitudes diverge, correlating with performance gaps on semantic benchmarks. It introduces a token-level Alignment Path Score (APS) and demonstrates that alignment quality across tokens and layers predicts the modality gap, especially under LoRA fine-tuning. By analyzing both coarse- and fine-grained representations and performing targeted interventions (angle projection and length normalization) on critical tokens, the work shows that improving token-level alignment can enhance speech input correctness. These findings provide theoretical and methodological guidance for optimizing cross-modal alignment in LSLMs and highlight directions for training-time integration to bridge the speech-text gap.

Abstract

End-to-end Large Speech Language Models (LSLMs) have demonstrated impressive conversational generation abilities, yet consistently fall short of traditional pipeline systems on semantic understanding benchmarks. In this work, we reveal through systematic experimentation that although LSLMs lose some text input performance after speech-text alignment training, the performance gap between speech and text inputs is more pronounced, which we refer to as the modality gap. To understand this gap, we analyze both coarse- and fine-grained text and speech representations. At the coarse-grained level, representations of speech and text in deeper layers are found to be increasingly aligned in direction (cosine similarity), while concurrently diverging in magnitude (Euclidean distance). We further find that representation similarity is strongly correlated with the modality gap. At the fine-grained level, a spontaneous token-level alignment pattern between text and speech representations is observed. Based on this, we introduce the Alignment Path Score to quantify token-level alignment quality, which exhibits stronger correlation with the modality gap. Building on these insights, we design targeted interventions on critical tokens through angle projection and length normalization. These strategies demonstrate the potential to improve correctness for speech inputs. Our study provides the first systematic empirical analysis of the modality gap and alignment mechanisms in LSLMs, offering both theoretical and methodological guidance for future optimization.

Figures (9)

• Figure 1: The LSLM architecture used in this study to analyze the speech–text modality gap.
• Figure 2: Overall Score (%, $\uparrow$) on the test set of 8 LSLMs at epoch 2 under different training methods and modality inputs. The red numbers on each bar show the score difference relative to the bar immediately to its left.
• Figure 3: Overview of coarse-grained speech-text representations computation.
• Figure 4: Layer-wise cosine similarity (blue) and Euclidean distance (orange) between speech and text representations. Each curve corresponds to a training checkpoint, with the horizontal axis indicating the model layers.
• Figure 5: Linear relationship between cross-modal similarity and modality performance gap ($\textcolor{beikeblue}{GAP}$).