FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding

TL;DR

The paper addresses cross-lingual transfer in multilingual NLP by proposing FILTER, a three-stage architecture that encodes source and translated text in shallow layers, fuses representations cross-lingually in intermediate layers, and re-encodes with language-specific deep layers. It introduces a self-teaching KL loss using auto-generated soft labels for translated target text to handle unreliable target-language labels, and demonstrates state-of-the-art performance on XTREME and XGLUE with significant reductions in cross-lingual transfer gaps. The key contributions are the FILTER architecture, the self-teaching loss, and extensive ablations showing when and how cross-language fusion most benefits various task categories. This work offers a robust, configurable framework for multilingual NLP that can adapt to classification, QA, POS, and NER tasks across many languages.

Abstract

Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and XLM, have achieved great success in cross-lingual representation learning. However, when applied to zero-shot cross-lingual transfer tasks, most existing methods use only single-language input for LM finetuning, without leveraging the intrinsic cross-lingual alignment between different languages that proves essential for multilingual tasks. In this paper, we propose FILTER, an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. Specifically, FILTER first encodes text input in the source language and its translation in the target language independently in the shallow layers, then performs cross-language fusion to extract multilingual knowledge in the intermediate layers, and finally performs further language-specific encoding. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. For simple tasks such as classification, translated text in the target language shares the same label as the source language. However, this shared label becomes less accurate or even unavailable for more complex tasks such as question answering, NER and POS tagging. To tackle this issue, we further propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language. Extensive experiments demonstrate that FILTER achieves new state of the art on two challenging multilingual multi-task benchmarks, XTREME and XGLUE.

Figures (3)

• Figure 1: Examples from XTREME for cross-lingual natural language inference, part-of-speech tagging, and question answering tasks. The source language is English; the target language can be any other languages.
• Figure 2: Comparison between different methods for finetuning XLM-R model for the XTREME benchmark. (a) Translate-train baseline. (b) Another baseline via simple concatenation of translated text. (c) Proposed Filter approach. (a) and (b) can be considered as special instantiations of Filter by setting $m=24, k=0$ and $m=0, k=24$, respectively.
• Figure 3: Results on the dev set of PAWS-X, POS and MLQA with different $m$ and $k$ values.