How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations

TL;DR

This work investigates how multimodal foundation models encode text and speech across 30 languages by quantifying cross-modal and cross-lingual alignment using SVCCA on activations from three architectures (Seamless, SONAR, SALMONN). It reveals that cross-modal representations converge with depth despite initial modality-specific layers, and that length adaptation helps reduce the modality gap mainly for higher-resource languages. Speech exhibits larger cross-lingual differences than text, and models not explicitly trained for modality-agnostic representations often show a modality gap larger than the language gap. The findings inform practical model selection and design, underscoring tokenizer limitations and data constraints, and recommending representation-focused analyses during model development to close modality and language gaps.

Abstract

Multimodal foundation models aim to create a unified representation space that abstracts away from surface features like language syntax or modality differences. To investigate this, we study the internal representations of three recent models, analyzing the model activations from semantically equivalent sentences across languages in the text and speech modalities. Our findings reveal that: 1) Cross-modal representations converge over model layers, except in the initial layers specialized at text and speech processing. 2) Length adaptation is crucial for reducing the cross-modal gap between text and speech, although current approaches' effectiveness is primarily limited to high-resource languages. 3) Speech exhibits larger cross-lingual differences than text. 4) For models not explicitly trained for modality-agnostic representations, the modality gap is more prominent than the language gap.

Figures (5)

• Figure 1: We use the similarity between model activations for the same sentences in different languages and modalities to measure language and modality gaps.
• Figure 2: Average cross-modal similarity over all languages over model layers. X-axis markers "in": input word embeddings or audio features, "len": after length adaptor in Seamless, "pool": after pooling in SONAR, "enc": after the frozen audio encoder, before length adaptation by window-level Q-Former in SALMONN.
• Figure 3: Average cross-lingual similarities between all language pairs in speech/text modality over model layers.
• Figure 4: To visually verify how the models progressively process language and modality gaps, we use 2D visualization with t-SNE JMLR:v9:vandermaaten08a for speech and text at a middle layer (14th, 14th, 18th from left to right). For Seamless and SONAR, texts are organized by semantics while speech remains clustered by language or language family. For SALMONN, languages with diverse scripts remain distinct in text representations.
• Figure 5: Given representations of text sentences at the last layer in one language, similarity to the same sentences in speech ("intra-lingual cross-modal"), their translations in text ("cross-lingual text"), and their translations in speech ("cross-lingual speech"). Latter two shown as range over all 29 language pairs. Language codes in \ref{['tab:language_statistics']}.