Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

Nadezhda Chirkova, Vassilina Nikoulina

TL;DR

The paper tackles zero-shot cross-lingual generation, a challenging setting where a model fine-tuned on English must generate in target languages. It conducts a unified, large-scale comparison of backbones (mT5, mBART, NLLB-200) and adaptation methods, with emphasis on hyperparameters and intermediate tuning. Key findings show that careful learning rate tuning combined with intermediate tuning makes simple full finetuning a strong baseline, often matching or exceeding more complex adaptation strategies and even approaching data-translation baselines. The work provides practical guidance on when and why certain methods help, highlighting model-specific effects and the potential for zero-shot generation to be practically competitive for multilingual tasks like XL-Sum and XQuAD.

Abstract

Zero-shot cross-lingual knowledge transfer enables a multilingual pretrained language model, finetuned on a task in one language, make predictions for this task in other languages. While being broadly studied for natural language understanding tasks, the described setting is understudied for generation. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work we compare various approaches proposed from the literature in unified settings, also including alternative backbone models, namely mBART and NLLB-200. We first underline the importance of tuning learning rate used for finetuning, which helps to substantially alleviate the problem of generation in the wrong language. Then, we show that with careful learning rate tuning, the simple full finetuning of the model acts as a very strong baseline and alternative approaches bring only marginal improvements. Finally, we find that mBART performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. Our final zero-shot models reach the performance of the approach based on data translation which is usually considered as an upper baseline for zero-shot cross-lingual transfer in generation.

Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

TL;DR

The paper tackles zero-shot cross-lingual generation, a challenging setting where a model fine-tuned on English must generate in target languages. It conducts a unified, large-scale comparison of backbones (mT5, mBART, NLLB-200) and adaptation methods, with emphasis on hyperparameters and intermediate tuning. Key findings show that careful learning rate tuning combined with intermediate tuning makes simple full finetuning a strong baseline, often matching or exceeding more complex adaptation strategies and even approaching data-translation baselines. The work provides practical guidance on when and why certain methods help, highlighting model-specific effects and the potential for zero-shot generation to be practically competitive for multilingual tasks like XL-Sum and XQuAD.

Abstract

Zero-shot cross-lingual knowledge transfer enables a multilingual pretrained language model, finetuned on a task in one language, make predictions for this task in other languages. While being broadly studied for natural language understanding tasks, the described setting is understudied for generation. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work we compare various approaches proposed from the literature in unified settings, also including alternative backbone models, namely mBART and NLLB-200. We first underline the importance of tuning learning rate used for finetuning, which helps to substantially alleviate the problem of generation in the wrong language. Then, we show that with careful learning rate tuning, the simple full finetuning of the model acts as a very strong baseline and alternative approaches bring only marginal improvements. Finally, we find that mBART performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. Our final zero-shot models reach the performance of the approach based on data translation which is usually considered as an upper baseline for zero-shot cross-lingual transfer in generation.
Paper Structure (29 sections, 12 figures, 4 tables)

This paper contains 29 sections, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology and experimental setup
  4. Adaptation methods.
  5. Models.
  6. Evaluation.
  7. Adaptation settings.
  8. Hyperparameter tuning.
  9. Experiments
  10. Effect of learning rate.
  11. Effect of intermediate tuning.
  12. Comparison of adaptation methods.
  13. Comparison of models.
  14. Comparison versus data translation.
  15. Experiments with larger models.
  16. ...and 14 more sections

Figures (12)

  • Figure 1: Learning rate plays a key role in cross-lingual transfer: decreasing LR almost completely eliminates generation in the wrong language with standard full finetuning, and often brings larger improvements that using complex adaptation methods developed to overcome this problem. Full results in Fig. \ref{['fig:mt5_xlsum']}--\ref{['fig:mbart_pt_xquad']} in Appendix.
  • Figure 2: Comparison of adaptation methods, with tuned learning rates and intermediate tuning when it is needed. Results averaged across target languages and 2 runs. Language correct rate is close to 100% in almost all cases, due to hyperparameter tuning. The exception is prompt tuning of mT5 in the XQuAD task which is not shown because of too low performance. Main conclusions: (1) Straightforward full finetuning is a strong approach which reaches or approaches the performance of data translation in all cases. (2) None of other approaches outperform full finetuning consistently in all cases: using several sources languages works well for mT5 but not for mBART and freezing decoder works well for mBART but not mT5. (3) One of zero-shot approaches reaches or outperforms a strong and computationally expensive baseline, data translation, in all cases.
  • Figure 3: Comparison of base models with full finetuning. Each plot averaged over 3 runs. Correct language rate is close to 100%, due to hyperparameter tuning, in almost all cases except the translation-tuned version of mBART. pt: pretrained version of mBART, tr: translation-finetuned version of mBART. Main conclusion: mBART and mT5 of similar sizes perform on par; NLLB performs well in summarization for Latin-alphabet languages.
  • Figure 4: Example predictions for a selection of models. Avg. len. over evaluation corpora in French, in characters. Red highlights errors or extra tokens.
  • Figure 5: Comparison of self-supervised objectives for intermediate tuning, with freezing decoder and embeddings as an adaptation method. Task metric: Rouge-2 for XL-Sum, F1 for XQuAD. Correct language rate is close to 100% in all cases except pretrained mBART on XL-Sum.
  • ...and 7 more figures