Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages

Tewodros Kederalah Idris, Prasenjit Mitra, Roald Eiselen

TL;DR

This study evaluates whether embedding similarity metrics can predict cross-lingual transfer performance for low-resource African languages. Using five metrics across three encoder-based models, 12 languages, and three NLP tasks, it analyzes 816 transfer experiments and compares against URIEL typology. The main findings show cosine_gap, P@1, and CSLS yield moderate predictive power ($\rho$ around $0.4$–$0.6$), while CKA is weak ($\rho$ near $0.1$); crucially, pooling across models produces Simpson's paradox, making model-stratified analyses essential. Practically, cosine_gap serves as a sensible default for guiding source-language selection, with model-specific validation and URIEL as a supplement when embedding signals are weak.

Abstract

Cross-lingual transfer is essential for building NLP systems for low-resource African languages, but practitioners lack reliable methods for selecting source languages. We systematically evaluate five embedding similarity metrics across 816 transfer experiments spanning three NLP tasks, three African-centric multilingual models, and 12 languages from four language families. We find that cosine gap and retrieval-based metrics (P@1, CSLS) reliably predict transfer success ($ρ= 0.4-0.6$), while CKA shows negligible predictive power ($ρ\approx 0.1$). Critically, correlation signs reverse when pooling across models (Simpson's Paradox), so practitioners must validate per-model. Embedding metrics achieve comparable predictive power to URIEL linguistic typology. Our results provide concrete guidance for source language selection and highlight the importance of model-specific analysis.

Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages

TL;DR

This study evaluates whether embedding similarity metrics can predict cross-lingual transfer performance for low-resource African languages. Using five metrics across three encoder-based models, 12 languages, and three NLP tasks, it analyzes 816 transfer experiments and compares against URIEL typology. The main findings show cosine_gap, P@1, and CSLS yield moderate predictive power ( around –), while CKA is weak ( near ); crucially, pooling across models produces Simpson's paradox, making model-stratified analyses essential. Practically, cosine_gap serves as a sensible default for guiding source-language selection, with model-specific validation and URIEL as a supplement when embedding signals are weak.

Abstract

Cross-lingual transfer is essential for building NLP systems for low-resource African languages, but practitioners lack reliable methods for selecting source languages. We systematically evaluate five embedding similarity metrics across 816 transfer experiments spanning three NLP tasks, three African-centric multilingual models, and 12 languages from four language families. We find that cosine gap and retrieval-based metrics (P@1, CSLS) reliably predict transfer success (), while CKA shows negligible predictive power (). Critically, correlation signs reverse when pooling across models (Simpson's Paradox), so practitioners must validate per-model. Embedding metrics achieve comparable predictive power to URIEL linguistic typology. Our results provide concrete guidance for source language selection and highlight the importance of model-specific analysis.
Paper Structure (45 sections, 4 equations, 1 figure, 19 tables)

This paper contains 45 sections, 4 equations, 1 figure, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cross-Lingual Transfer and Source Language Selection
  4. Embedding Similarity Metrics
  5. Linguistic Typology for Transfer Prediction
  6. African Language NLP
  7. Methods
  8. Embedding Similarity Metrics
  9. Magnitude-Based Metrics
  10. Retrieval-Based Metrics
  11. Structural Metrics
  12. Correlation Analysis Framework
  13. Linguistic Baseline: URIEL Distances
  14. Experimental Setup
  15. Models
  16. ...and 30 more sections

Figures (1)

  • Figure 1: Simpson's Paradox in metric-transfer correlations. (a) Pooling across models suggests a -ve relationship ($\rho = -0.18$). (b) Stratifying by model reveals +ve correlations in each model. Points show all 396 NER language-pair-model combinations. Panel (a) displays points in gray with a pooled OLS regression line; Panel (b) colors points by model with separate regression lines. Legend shows per-model Spearman correlations.