Revisiting Modality Invariance in a Multilingual Speech-Text Model via Neuron-Level Analysis

TL;DR

This work interrogates whether a multilingual speech–text model achieves modality-invariant language representations. Using neuron-level average-precision ranking, inference-time median-replacement interventions, and activation-magnitude analyses across the speech encoder, text encoder, and shared decoder in SeamlessM4T v2, the authors uncover partial modality invariance. Encoders push toward language-agnostic representations, but the shared decoder struggles to recover language identity when inputs differ in modality, particularly in speech-conditioned decoding; modality-specific structure is sharply localized to cross-attention projections. Activation magnitudes reveal that speech decoding and non-dominant scripts rely on a small subset of highly active neurons, suggesting localized, potentially brittle cross-modal routing. Overall, cross-attention emerges as a bottleneck for modality reconciliation, motivating strategies to improve cross-modal encoder alignment to enable more robust modality-invariant processing.

Abstract

Multilingual speech-text foundation models aim to process language uniformly across both modality and language, yet it remains unclear whether they internally represent the same language consistently when it is spoken versus written. We investigate this question in SeamlessM4T v2 through three complementary analyses that probe where language and modality information is encoded, how selective neurons causally influence decoding, and how concentrated this influence is across the network. We identify language- and modality-selective neurons using average-precision ranking, investigate their functional role via median-replacement interventions at inference time, and analyze activation-magnitude inequality across languages and modalities. Across experiments, we find evidence of incomplete modality invariance. Although encoder representations become increasingly language-agnostic, this compression makes it more difficult for the shared decoder to recover the language of origin when constructing modality-agnostic representations, particularly when adapting from speech to text. We further observe sharply localized modality-selective structure in cross-attention key and value projections. Finally, speech-conditioned decoding and non-dominant scripts exhibit higher activation concentration, indicating heavier reliance on a small subset of neurons, which may underlie increased brittleness across modalities and languages.

Figures (53)

• Figure 1: Overview of our method (length adapter omitted for clarity). We identify language-specific neurons via Average Precision (AP) ranking cuadros2022selfkojima-etal-2024-multilingual and categorize selected neurons into four groups based on how labels are constructed. After recording activations during inference, we average them across the token dimension $N$. If the activation of a given neuron reliably predicts a particular feature (by AP), we regard that neuron as specialized.
• Figure 2: top-1000 selective neurons identified under different conditions. Colors indicate the model subcomponents from which the neurons originate. (a--d) Unimodal language-specific neurons: neurons selective for German within a single modality. (a--b) Neurons identified for text German in the encoder (attention and FFN). (c) Neurons identified for speech German in the decoder (attention). (d) Neurons identified for text German in the decoder (attention). (e--f) Modality-specific neurons in the decoder, highlighting units that distinguish speech from text input: (e) speech-specific and (f) text-specific neurons.
• Figure 3: Decoder overlap (bottom-1000 neurons) between speech-conditioned and text-conditioned unimodal language-specific neurons across languages.
• Figure 4: Unimodal language-specific neurons in the speech encoder for German speech-to-text translation. Shown are the bottom-1000 and top-1000 neurons ranked by language selectivity, across FFN, attention, and convolutional submodules. Colors correspond to different subcomponents. Language-specific neurons emerge in early layers, peak in mid layers, and decrease toward higher layers.
• Figure 6: Deviation from the mean activation trend (mean over S2T and T2T) across decoder layers, shown by language and modality. It is scaled by a factor of 1000 for clearer visualiation.