LangGPS: Language Separability Guided Data Pre-Selection for Joint Multilingual Instruction Tuning

TL;DR

This paper addresses the data sensitivity of joint multilingual instruction tuning by introducing LangGPS, a two-stage data pre-selection framework guided by language separability measured in the model's representation space. It combines a lightweight separability-based filter with existing data selection methods, enabling principled integration with diversity, task relevance, and gradient-based selectors. Empirical results across six benchmarks and 22 languages show LangGPS consistently improves multilingual performance, especially for understanding tasks and low-resource languages, and reveal complementary roles of high- and low-separability samples. Additionally, language separability informs multilingual curriculum learning, where interleaving samples of diverse separability levels yields stable, generalizable gains, offering a linguistically informed pathway to enhance multilingual LLMs.

Abstract

Joint multilingual instruction tuning is a widely adopted approach to improve the multilingual instruction-following ability and downstream performance of large language models (LLMs), but the resulting multilingual capability remains highly sensitive to the composition and selection of the training data. Existing selection methods, often based on features like text quality, diversity, or task relevance, typically overlook the intrinsic linguistic structure of multilingual data. In this paper, we propose LangGPS, a lightweight two-stage pre-selection framework guided by language separability which quantifies how well samples in different languages can be distinguished in the model's representation space. LangGPS first filters training data based on separability scores and then refines the subset using existing selection methods. Extensive experiments across six benchmarks and 22 languages demonstrate that applying LangGPS on top of existing selection methods improves their effectiveness and generalizability in multilingual training, especially for understanding tasks and low-resource languages. Further analysis reveals that highly separable samples facilitate the formation of clearer language boundaries and support faster adaptation, while low-separability samples tend to function as bridges for cross-lingual alignment. Besides, we also find that language separability can serve as an effective signal for multilingual curriculum learning, where interleaving samples with diverse separability levels yields stable and generalizable gains. Together, we hope our work offers a new perspective on data utility in multilingual contexts and support the development of more linguistically informed LLMs.

Figures (5)

• Figure 1: The left shows the t-SNE visualization of samples selected by different strategies under same language distribution; each color denotes one of 31 languages. The representations are last-token hidden states from LLaMA-3.1-8B. On the right, panel (a) presents the performance of LLaMA-3.1-8B after training with the three types of samples on MMMLU across training sizes; while panel (b) illustrates the pairwise similarity distributions of selected data. (See detailed settings in Appendix \ref{['app:pre_exp']})
• Figure 2: Average relative gains or declines (in percentage) when applying LangGPS on top of Rand and the three strongest-performing baselines, shown separately for high-resource and low-resource languages.
• Figure 3: Effect of the pre-selection ratio $\rho$ on performance (MMMLU and XLSum, 5% setting, LLaMA-3.1-8B). We vary $\rho$ from 10% to 100% and report results using both Rand and LESS as downstream selectors.
• Figure 4: Average translation performance of LLaMA-3.1-8B on FLORES+ ($X \rightarrow En$ and $En \rightarrow X$, where $X$ = De, Fr, Zh) under varying training sizes, comparing models trained on top separability-scoring, bottom separability-scoring, and randomly selected samples.
• Figure 5: t-SNE visualizations of output representations from LLaMA-3.1-8B under different training settings: before SFT, after SFT with Rand-5%, LESS-5%, LangGPS+Rand-5%, and LangGPS+LESS-5%. Average silhouette scores are also reported.