Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Choosing Transfer Languages for Cross-Lingual Learning

Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig

TL;DR

This work reframes cross-lingual transfer language selection as a learning-to-rank problem, introducing LangRank which combines dataset-dependent features (e.g., dataset size, lexical overlap) with dataset-independent typological distances (via URIEL) to predict top transfer languages for a target low-resource language. Using a gradient-boosted decision-tree model with LambdaRank, LangRank is trained on exhaustive transfer-language evaluations across four NLP tasks (MT, EL, POS tagging, DEP) and evaluated with leave-one-out cross-validation, outperforming single-feature heuristics in NDCG@3 while offering interpretable feature importance insights. The results show that dataset features dominate some tasks (notably MT) while typological distances remain valuable across tasks, and that LangRank can provide strong guidance even when only linguistic features are available. Practically, this reduces the search and data-collection burden for cross-lingual transfer and yields actionable rules of thumb for transfer-language choice, with broad implications for efficient multilingual NLP development.

Abstract

Cross-lingual transfer, where a high-resource transfer language is used to improve the accuracy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low-resource languages. However, given a particular task language, it is not clear which language to transfer from, and the standard strategy is to select languages based on ad hoc criteria, usually the intuition of the experimenter. Since a large number of features contribute to the success of cross-lingual transfer (including phylogenetic similarity, typological properties, lexical overlap, or size of available data), even the most enlightened experimenter rarely considers all these factors for the particular task at hand. In this paper, we consider this task of automatically selecting optimal transfer languages as a ranking problem, and build models that consider the aforementioned features to perform this prediction. In experiments on representative NLP tasks, we demonstrate that our model predicts good transfer languages much better than ad hoc baselines considering single features in isolation, and glean insights on what features are most informative for each different NLP tasks, which may inform future ad hoc selection even without use of our method. Code, data, and pre-trained models are available at https://github.com/neulab/langrank

Choosing Transfer Languages for Cross-Lingual Learning

TL;DR

This work reframes cross-lingual transfer language selection as a learning-to-rank problem, introducing LangRank which combines dataset-dependent features (e.g., dataset size, lexical overlap) with dataset-independent typological distances (via URIEL) to predict top transfer languages for a target low-resource language. Using a gradient-boosted decision-tree model with LambdaRank, LangRank is trained on exhaustive transfer-language evaluations across four NLP tasks (MT, EL, POS tagging, DEP) and evaluated with leave-one-out cross-validation, outperforming single-feature heuristics in NDCG@3 while offering interpretable feature importance insights. The results show that dataset features dominate some tasks (notably MT) while typological distances remain valuable across tasks, and that LangRank can provide strong guidance even when only linguistic features are available. Practically, this reduces the search and data-collection burden for cross-lingual transfer and yields actionable rules of thumb for transfer-language choice, with broad implications for efficient multilingual NLP development.

Abstract

Cross-lingual transfer, where a high-resource transfer language is used to improve the accuracy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low-resource languages. However, given a particular task language, it is not clear which language to transfer from, and the standard strategy is to select languages based on ad hoc criteria, usually the intuition of the experimenter. Since a large number of features contribute to the success of cross-lingual transfer (including phylogenetic similarity, typological properties, lexical overlap, or size of available data), even the most enlightened experimenter rarely considers all these factors for the particular task at hand. In this paper, we consider this task of automatically selecting optimal transfer languages as a ranking problem, and build models that consider the aforementioned features to perform this prediction. In experiments on representative NLP tasks, we demonstrate that our model predicts good transfer languages much better than ad hoc baselines considering single features in isolation, and glean insights on what features are most informative for each different NLP tasks, which may inform future ad hoc selection even without use of our method. Code, data, and pre-trained models are available at https://github.com/neulab/langrank

Paper Structure

This paper contains 28 sections, 9 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Ranking Features
  4. Data-dependent Features
  5. Dataset Size:
  6. Type-Token Ratio (TTR):
  7. Word Overlap and Subword Overlap:
  8. Dataset-independent Features
  9. Geographic distance ($d_{geo}$):
  10. Genetic distance ($d_{gen}$):
  11. Inventory distance ($d_{inv}$):
  12. Syntactic distance ($d_{syn}$):
  13. Phonological distance ($d_{pho}$):
  14. Featural distance ($d_{fea}$):
  15. Ranking Model
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Workflow of learning to select the transfer languages for an NLP task: (1) train a set of NLP models with all available transfer languages and collect evaluation scores, (2) train a ranking model to predict the top transfer languages.
  • Figure 2: The best evaluation score (BLEU for MT, accuracy for EL and POS, and LAS for DEP) attainable by trying out the top $K$ transfer languages recommended by the LangRank models and the single-feature baselines.
  • Figure 3: Normalized feature importance for the MT, EL, POS and DEP tasks.
  • Figure 4: An example of the decision tree learned in the machine translation task for Galician as task language.