Analyzing the Effect of Linguistic Similarity on Cross-Lingual Transfer: Tasks and Experimental Setups Matter

TL;DR

The paper addresses how linguistic similarity influences cross-lingual transfer across a broad, real-world set of languages and tasks. It implements zero-shot transfer for POS tagging, parsing, and topic classification using extensive datasets and a diverse suite of similarity measures, revealing that the most predictive metrics depend on the task and input representation. Syntactic similarity best predicts parsing/POS performance, while trigram overlap dominates topic classification; training size and some phonological measures are weak predictors, and transliteration or pretraining data can alter transfer patterns. The authors provide practical guidance for choosing source languages and demonstrate that cross-task signals can inform decisions when task-specific similarity information is unavailable. This work highlights the need for large-scale, multi-task analyses to generalize cross-lingual transfer insights and suggests future work should broaden languages, tasks, and modeling approaches.

Abstract

Cross-lingual transfer is a popular approach to increase the amount of training data for NLP tasks in a low-resource context. However, the best strategy to decide which cross-lingual data to include is unclear. Prior research often focuses on a small set of languages from a few language families and/or a single task. It is still an open question how these findings extend to a wider variety of languages and tasks. In this work, we analyze cross-lingual transfer for 263 languages from a wide variety of language families. Moreover, we include three popular NLP tasks: POS tagging, dependency parsing, and topic classification. Our findings indicate that the effect of linguistic similarity on transfer performance depends on a range of factors: the NLP task, the (mono- or multilingual) input representations, and the definition of linguistic similarity.

Figures (12)

• Figure 1: Languages included in our experiments. Green indicates languages included in all tasks, blue languages only used for POS tagging and parsing, and purple languages only used for topic classification. Base map via naturalearthdata.com (CC0).
• Figure 2: Different experiments produce different transfer patterns. NLP transfer results for all combinations of training (columns) and test languages (rows). The darker a cell, the higher the score. The three heatmaps for the grammatical tasks are sorted in the same order, and the three heatmaps with the topic classification results are sorted in the same order. The darker diagonal shows the within-language scores. Large, labelled heatmaps are in Appendix §\ref{['sec:appendix-heatmaps']}.
• Figure 3: Mean correlation scores between task results and similarity measures. "Word*" = overlap between words (wor; UD tasks) trigrams (tri; topics-base/translit), and subword tokens (swt; topics-mbert). Dotted lines are added for intelligibility.
• Figure 4: Left: Relationship between phylogenetic and syntactic similarity -- unrelated or distantly related languages can be syntactically similar or dissimilar, but all closely related languages are syntactically similar. Right: Relationship between character overlap (between training and test sets) and the classification scores of the topics-base model -- transfer between languages with low character overlap works poorly, but high overlap does not guarantee good transfer.
• Figure 5: Differences between the performance on the original test treebanks and their transliterated counterparts for POS tagging (top) and parsing (LAS, bottom). Rows are for test sets, columns for training sets. Pink cells mark configurations where scores are better on the original data; green where scores are better on the transliterated treebank. Original writing systems are colour-coded. Writing systems in grey appear only for one language.