Exploring Alignment in Shared Cross-lingual Spaces

TL;DR

This paper addresses how multilingual contextualized embeddings align across languages by proposing an unsupervised latent-concept analysis. It discovers concepts via layerwise clustering and introduces CALIGN and COLAP to quantify concept alignment and cross-language overlap, respectively, applying them to mT5, mBERT, and XLM-R across MT, NER, and SST-2 with multiple languages. Key findings show that deeper layers harbor language-agnostic semantic concepts, fine-tuning further calibrates latent spaces to enhance cross-lingual alignment (including zero-shot transfer), and encoder-decoder spaces exhibit distinct, language-specific tendencies in seq2seq tasks. The work offers a latent-space perspective on multilingual transfer, with practical implications for designing models that better support zero-shot and low-resource languages. The analysis highlights that alignment is more robust for closely related languages and that task-driven calibration can partly explain zero-shot capabilities, pointing to actionable directions for improving multilingual NLP systems.

Abstract

Despite their remarkable ability to capture linguistic nuances across diverse languages, questions persist regarding the degree of alignment between languages in multilingual embeddings. Drawing inspiration from research on high-dimensional representations in neural language models, we employ clustering to uncover latent concepts within multilingual models. Our analysis focuses on quantifying the \textit{alignment} and \textit{overlap} of these concepts across various languages within the latent space. To this end, we introduce two metrics \CA{} and \CO{} aimed at quantifying these aspects, enabling a deeper exploration of multilingual embeddings. Our study encompasses three multilingual models (\texttt{mT5}, \texttt{mBERT}, and \texttt{XLM-R}) and three downstream tasks (Machine Translation, Named Entity Recognition, and Sentiment Analysis). Key findings from our analysis include: i) deeper layers in the network demonstrate increased cross-lingual \textit{alignment} due to the presence of language-agnostic concepts, ii) fine-tuning of the models enhances \textit{alignment} within the latent space, and iii) such task-specific calibration helps in explaining the emergence of zero-shot capabilities in the models.\footnote{The code is available at \url{https://github.com/baselmousi/multilingual-latent-concepts}}

Figures (30)

• Figure 1: Overview of CALIGN and COLAP metrics in latent spaces of multilingual models, and how the space re-calibrates after fine-tuning. The top row shows concepts learned in mT5 across different languages: (a) English (b) German, (c) Spanish, (d) Arabic.
• Figure 2: Quantifying Concept Alignment CALIGN (%) in German–English Concepts: Dotted lines depict base models, while solid lines represent fine-tuned models across different multilingual models.
• Figure 3: Lower layers capture lexical concepts (a,b), while higher layers focus on semantic concepts (c,d).
• Figure 4: Concept Alignment (%) in mT5. Dotted lines represent base models, solid lines denote fine-tuned French–English MT models, and dashed lines depict zero-shot alignment for German–English and Spanish–English.
• Figure 5: Concept Alignment (%) in mBERT. Solid lines: fine-tuned German–English NER model. Dashed lines: zero-shot alignment for French and Spanish.