How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning

TL;DR

This work investigates cross-lingual data sharing in multilingual fine-tuning from a data-centric perspective using TracIn, across XLM-R fine-tuned on XNLI, PAWS-X, and MARC with five languages. It shows that multilingual models rely on training data from multiple languages from early in fine-tuning, with reliance growing over epochs, and that cross-lingual sharing can be reinforcing or complementary depending on the task. The study demonstrates substantial cross-language influence in both parallel and non-parallel datasets and highlights differences between zero-shot and seen-language fine-tuning scenarios. It also analyzes the impact of data imbalance and provides a framework for tracing influential training samples, offering practical guidance for multilingual data curation and fine-tuning strategies.

Abstract

Multilingual large language models (MLLMs) are jointly trained on data from many different languages such that representation of individual languages can benefit from other languages' data. Impressive performance on zero-shot cross-lingual transfer shows that these models are capable of exploiting data from other languages. Yet, it remains unclear to what extent, and under which conditions, languages rely on each other's data. In this study, we use TracIn (Pruthi et al., 2020), a training data attribution (TDA) method, to retrieve the most influential training samples seen during multilingual fine-tuning for a particular test language. This allows us to analyse cross-lingual sharing mechanisms of MLLMs from a new perspective. While previous work studied cross-lingual sharing at the level of model parameters, we present the first approach to study cross-lingual sharing at the data level. We find that MLLMs rely on data from multiple languages from the early stages of fine-tuning and that this reliance gradually increases as fine-tuning progresses. We further study how different fine-tuning languages influence model performance on a given test language and find that they can both reinforce and complement the knowledge acquired from data of the test language itself.

Figures (13)

• Figure 1: Average percentage of decrease in model confidence across test samples and fine-tuning languages when removing the top $k$ most positively influential training samples for PAWS-X.
• Figure 2: For each test language, we show the percentage of samples that each fine-tuning language contributed to the top 100 most positively (left) and negatively (right) influential training samples averaged across all test samples.
• Figure 3: For each test language, we plot the percentage of samples coming from their own language that were included in the most positively influential training samples, i.e. the extent to which the model relies on its own language and how this changes over fine-tuning epochs.
• Figure 4: Percentage of samples that each fine-tuning language contributed to the top 100 most influential samples for Korean and Spanish during zero-shot testing.
• Figure 5: The percentage of data contributing to either the most positively (left) or negatively (right) influential samples for a particular language when adding $p$ % of data on top of that language's data during fine-tuning.