Analyzing and Improving Cross-lingual Knowledge Transfer for Machine Translation

TL;DR

This work addresses the challenge of cross-lingual knowledge transfer in machine translation by introducing Representational Transfer Potential (RTP) to quantify language similarity at the cross-attention level and predict translation gains. It proposes algorithmic methods (auxiliary similarity loss, cross-lingual/multilingual datastores, linear mapping) and retrieval-based approaches (multilingual kNN-MT) to boost low-resource translation while maintaining efficiency. It also investigates the fine-tuning of large language models, revealing a paradox where increased parallel data enhances general translation quality but degrades several emergent LLM capabilities; a mixed data fine-tuning approach mitigates these losses. Finally, it demonstrates that increasing language diversity during fine-tuning generally improves translation quality and cross-lingual generalization, with middle-layer representations showing the most significant adaptations, though gains plateau beyond a diversity threshold. Collectively, the thesis provides both theoretical insights and practical methods for more inclusive, robust multilingual MT systems, spanning MNMT and LLM-based translation paradigms.

Abstract

Multilingual machine translation systems aim to make knowledge accessible across languages, yet learning effective cross-lingual representations remains challenging. These challenges are especially pronounced for low-resource languages, where limited parallel data constrains generalization and transfer. Understanding how multilingual models share knowledge across languages requires examining the interaction between representations, data availability, and training strategies. In this thesis, we study cross-lingual knowledge transfer in neural models and develop methods to improve robustness and generalization in multilingual settings, using machine translation as a central testbed. We analyze how similarity between languages influences transfer, how retrieval and auxiliary supervision can strengthen low-resource translation, and how fine-tuning on parallel data can introduce unintended trade-offs in large language models. We further examine the role of language diversity during training and show that increasing translation coverage improves generalization and reduces off-target behavior. Together, this work highlights how modeling choices and data composition shape multilingual learning and offers insights toward more inclusive and resilient multilingual NLP systems.

Figures (26)

• Figure 1: Average cosine similarities between context vectors (see Equation \ref{['ch3:eq:xsim']}) for different source language combinations into English. Train data size is shown between brackets. The higher the similarity, the higher the degree of language invariance.
• Figure 2: Cross-attention similarities for all language combinations. Training data size into English depicted between brackets.
• Figure 3: The x-axis represents Representational Transfer Potentials (RTP), which measure the total transfer potential for a language (as detailed in Equation \ref{['ch3:eq:rtp']}), on FLORES-101. The y-axis illustrates the difference in BLEU scores (multilingual BLEU - bilingual BLEU) on FLORES-101. The size of the dots indicates the bilingual BLEU score. The correlation coefficient (Spearman's $\rho$) is $.77$ and it is statistically significant ($p<0.001$). The trend illustrates that a higher RTP value is positively associated with changes in translation performance in a multilingual setting.
• Figure 4: The x-axis represents Representational Transfer Potentials (RTP), which measure the total transfer potential for a language (as detailed in Equation \ref{['ch3:eq:rtp']}), on NTREX-128. The y-axis illustrates the difference in BLEU scores (multilingual BLEU - bilingual BLEU) on NTREX-128. The size of the dots indicates the bilingual BLEU score.
• Figure 5: Feature importance for transfer prediction: linear regression sign of coefficients. Absolute values are plotted. Black line indicates negative coefficient (e.g., genetic distance is negative).