ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

TL;DR

This work tackles the limitation of parallel corpora in cross-lingual pre-training by introducing ERNIE-M, which aligns cross-lingual semantics using monolingual data. It introduces two objectives: CAMLM for cross-lingual semantic alignment on parallel data and BTMLM to learn from monolingual corpora via pseudo-parallel token generation, building on an XLM-R foundation. The approach achieves state-of-the-art results across XNLI, MLQA, CoNLL, PAWS-X, and Tatoeba, with ablations confirming the effectiveness of CAMLM and BTMLM. The method significantly reduces reliance on large parallel corpora, enabling robust multilingual representations and improved transfer, especially for low-resource languages.

Abstract

Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for low-resource languages. In this paper, we propose ERNIE-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks.

Figures (6)

• Figure 1: Overview of MMLM, TLM and CAMLM training. The input sentences in sub-figure (a) are monolingual sentences; $x$ and $y$ represent monolingual input sentences in different languages. The input sentences in sub-figures (b) and (c) are parallel sentences; $x$ and $y$ denote the source and target sentences of the parallel sentences, respectively. $h$ indicates the token predicted by the model.
• Figure 2: Self-attention mask matrix in MMLM, TLM and CAMLM. We use different self-attention masks for different pre-training objectives.
• Figure 3: Overview of BTMLM training; the left figure represents the first stage of BTMLM, predicting the pseudo-tokens. The right figure represents the second stage of the BTMLM, making predictions based on the predicted pseudo-tokens and original sentences.
• Figure 4: Self-attention matrix of BTMLM Stage 1.
• Figure 5: Tatoeba results for each language. The languages are sorted according to their size in the pre-trained corpus from smallest to largest. Fine-tuning can significantly improve the accuracy of different language families in the cross-lingual retrieval task.