Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies

TL;DR

The paper addresses how cross-lingual transfer from high-resource to low-resource languages is hindered by multiple intertwined factors. It introduces a controlled transfer framework that systematically transforms English data into t-English variants along single axes and evaluates two transfer regimes. The key finding is that embedding/word identity alignment is the dominant bottleneck, with syntactic shifts being much more recoverable when vocabulary remains aligned, and tokenizer quality having limited effect in the low-data regime. This framework helps researchers prioritize embedding alignment and vocabulary grounding when designing cross-lingual transfer scenarios.

Abstract

When we transfer a pretrained language model to a new language, there are many axes of variation that change at once. To disentangle the impact of different factors like syntactic similarity and vocabulary similarity, we propose a set of controlled transfer studies: we systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time, and then measure the resulting drops in a pretrained model's downstream performance. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment and embedding matrix re-initialization, even with continued pretraining on 15 million tokens. %On the other hand, transferring to a dataset with an unaligned vocabulary is extremely hard to recover from in the low-data regime. Moreover, good-quality tokenizers in the transfer language do not make vocabulary alignment easier. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.

Figures (7)

• Figure 1: Controlled transfer studies paradigm. We systematically transform GLUE tasks (t-GLUE) to target one linguistic factor, then finetune a pretrained language model on that dataset. The resulting drop in performance indicates the importance of that factor to crosslingual transfer. See \ref{['tab:badglish-examples']} for the list of transformations.
• Figure 2: Models are largely able to adapt to syntactic shifts with minor drops in performance. Averaged GLUE scores for t-Englishes with syntactic shifts. Realistic syntactic shifts slightly impact downstream performance, while reverse and random order impact performance more significantly. Error bars represent 95% confidence intervals over 3 random seeds. Results are depicted for RoBERTa, but are consistent for all 3 models that we tested: RoBERTa, DeBERTa, and XLM-R (all results in Figure \ref{['fig:all-models_syntactic']} in Appendix \ref{['sec:other_models']}).
• Figure 3: Token embedding transformations are hard to recover from, regardless of tokenizer. Averaged GLUE scores for t-Englishes with word identity perturbations. Any embedding reinitialization or shuffling, regardless of the tokenizer ultimately used, has a drastic effect on downstream performance. Error bars represent 95% confidence intervals over 3 random seeds. Results are depicted for RoBERTa, but are consistent for all 3 models that we tested: RoBERTa, DeBERTa, and XLM-R(all results in Figure \ref{['fig:all-models_tok']} in Appendix \ref{['sec:other_models']}).
• Figure 4: Our findings generalize to fine-tuning on non-English datasets. Fine-tuning on three different XNLI datasets yields similar findings the English GLUE findings: models can recover from the most extreme syntactic case (random ordering) much more effectively than from any of the embeddings-related perturbations. This indicates that our findings are not related to properties specific to the English language.
• Figure 5: Models are largely able to adapt to syntactic shifts with minor drops in performance. Results for the embedding transformations shown for RoBERTa in Figure \ref{['fig:perf-syn-shifts']}, for all models that we tested: RoBERTa, DeBERTa, and XLM-R.