Frustratingly Easy Label Projection for Cross-lingual Transfer

TL;DR

Experimental results show that the optimized version of mark-then-translate, which the authors call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods.

Abstract

Translating training data into many languages has emerged as a practical solution for improving cross-lingual transfer. For tasks that involve span-level annotations, such as information extraction or question answering, an additional label projection step is required to map annotated spans onto the translated texts. Recently, a few efforts have utilized a simple mark-then-translate method to jointly perform translation and projection by inserting special markers around the labeled spans in the original sentence. However, as far as we are aware, no empirical analysis has been conducted on how this approach compares to traditional annotation projection based on word alignment. In this paper, we present an extensive empirical study across 57 languages and three tasks (QA, NER, and Event Extraction) to evaluate the effectiveness and limitations of both methods, filling an important gap in the literature. Experimental results show that our optimized version of mark-then-translate, which we call EasyProject, is easily applied to many languages and works surprisingly well, outperforming the more complex word alignment-based methods. We analyze several key factors that affect the end-task performance, and show EasyProject works well because it can accurately preserve label span boundaries after translation. We will publicly release all our code and data.

Figures (5)

• Figure 1: Two methods for translating and projecting English ACE event triggers and named entities to Chinese. (a) Pipeline method based on word alignment: starting with machine translation of the English sentence to Chinese, followed by word-to-word alignment. Then, labeled spans are projected using heuristics. (b) Mark-then-translate: markers are inserted around the annotated spans in the text. The modified sentence with markers inserted is then fed as input to an MT system, projecting the label span markers to the target sentence as a byproduct of translation.
• Figure 2: Comparison of translation quality (x-axis) and end-task performance (y-axis) for different label projection methods on the WikiANN dataset using NLLB translation system. EasyProject (§\ref{['sec:easyproject_intro']}) outperforms alignment-based approach on F$_1$ scores for most languages, though inserting markers degrades translation quality. The experimental setting is detailed in § \ref{['sec:comparsion_to_alignment']}.
• Figure 3: The changes of projection rate and translation quality (measured by BLEU score) as fine-tuning more steps. We found 200 steps are sufficient to boost the projection rate while maintaining translation quality.
• Figure 4: NER $\Delta$F$_1$ (EasyProject+GMT over Fine-tune$_{\text{en}}$) vs. amount of pre-training data (GB) for XLM-RoBERTa$_{\text{large}}$.
• Figure 5: (a) NER F$_1$ for fine-tuning on different source and target languages. (b) The relative difference of F$_1$ for models trained on source and projected data together over source data only, when using EasyProject with the NLLB translation system. EasyProject can be used for transfer learning from any language to any language provided with the availability of multilingual MT systems.