Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

TL;DR

Unicoder introduces a universal language encoder pre-trained with five cross-lingual objectives to learn language-agnostic representations. By combining three novel cross-lingual pre-training tasks with MLM and TLM, and applying a multi-language fine-tuning strategy, it achieves state-of-the-art results on XNLI and XQA across multiple languages. The approach demonstrates that multi-task cross-lingual pre-training and joint multilingual fine-tuning yield robust cross-language transfer, with notable gains over prior baselines. This work advances practical cross-lingual NLP by enabling high performance in low-resource languages through training data in other languages.

Abstract

We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM, three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.

Figures (2)

• Figure 1: Unicoder consists of three cross-lingual pre-training tasks: (a) The cross-lingual word recovery model is to learn word relation from different languages (b) The cross-lingual paraphrase classification is to classify whether two sentences from different languages are paraphrase. (c) The cross-lingual masked language model is to train masked language model with cross-lingual document.
• Figure 2: Currently cross-lingual fine-tuning has three baseline approaches, they could be defined based on their training data and test data. Suppose we target to test on Chinese data, Translate-train is to train on Chinese training data which is translated from English and test on Chinese test data; Translate-Test is to train on English training data and test on English test data which is translated from Chinese; Cross-lingual test is to train in English training data and test on Chinese test data. Multi-language fine-tuning is to train on English training data and multiple other languages training data which are translated from English, then test on Chinese Test data.