Emerging Cross-lingual Structure in Pretrained Language Models

TL;DR

The paper challenges the assumption that cross-lingual transfer in multilingual masked language models relies on shared vocabularies or domain similarity. Through extensive ablations, it shows that sharing top-layer parameters suffices for strong cross-lingual performance, and that language-universal representations emerge even when anchor points are absent. It further demonstrates that independently trained monolingual BERTs can be aligned across languages via simple linear mappings, offering a plausible explanation for robust cross-lingual transfer. The work combines rigorous pretraining, targeted evaluations, and analyses of representation similarity to illuminate the structure of multilingual embedding spaces and guides future cross-lingual adaptation.

Abstract

We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of several factors that influence why these models are so effective for cross-lingual transfer. We show, contrary to what was previously hypothesized, that transfer is possible even when there is no shared vocabulary across the monolingual corpora and also when the text comes from very different domains. The only requirement is that there are some shared parameters in the top layers of the multi-lingual encoder. To better understand this result, we also show that representations from independently trained models in different languages can be aligned post-hoc quite effectively, strongly suggesting that, much like for non-contextual word embeddings, there are universal latent symmetries in the learned embedding spaces. For multilingual masked language modeling, these symmetries seem to be automatically discovered and aligned during the joint training process.

Figures (8)

• Figure 1: On the impact of anchor points and parameter sharing on the emergence of multilingual representations. We train bilingual masked language models and remove parameter sharing for the embedding layers and first few Transformers layers to probe the impact of anchor points and shared structure on cross-lingual transfer.
• Figure 2: Probing the layer similarity of monolingual BERT models. We investigate the similarity of separate monolingual BERT models at different levels. We use an orthogonal mapping between the pooled representations of each model. We also quantify the similarity using the centered kernel alignment (CKA) similarity index.
• Figure 3: Cross-lingual transfer of bilingual MLM on three tasks and language pairs under different settings. Others tasks and languages pairs follows similar trend. See \ref{['tab:all']} for full results.
• Figure 4: Alignment of word-level representations from monolingual BERT models on subset of MUSE benchmark. \ref{['fig:align-noncontextual-word']} and \ref{['fig:align-contextual-word']} are not comparable due to different embedding vocabularies.
• Figure 5: Contextual representation alignment of different layers for zero-shot cross-lingual transfer.