Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models

TL;DR

This work addresses why multilingual language models exhibit uneven performance across languages and tasks by analyzing 12 model- and language-centered features beyond data quantity and model size. It leverages Flores-200 and SIB-200 across 204 languages, applying regression models and SHAP attribution to quantify feature importance, with ensemble learners (e.g., Random Forest, Gradient Boosting, XGBoost) delivering strong predictive accuracy ($R^2$ up to ~0.9). Key findings include token similarity and country similarity as critical drivers, alongside pre-training data percentage and model size, while resource-related features show limited direct impact. The results offer practical guidance for equitable MLLM development, emphasizing cross-lingual token alignment and sociolinguistic context to improve performance for underrepresented languages.

Abstract

Multilingual language models (MLLMs) are crucial for handling text across various languages, yet they often show performance disparities due to differences in resource availability and linguistic characteristics. While the impact of pre-train data percentage and model size on performance is well-known, our study reveals additional critical factors that significantly influence MLLM effectiveness. Analyzing a wide range of features, including geographical, linguistic, and resource-related aspects, we focus on the SIB-200 dataset for classification and the Flores-200 dataset for machine translation, using regression models and SHAP values across 204 languages. Our findings identify token similarity and country similarity as pivotal factors, alongside pre-train data and model size, in enhancing model performance. Token similarity facilitates cross-lingual transfer, while country similarity highlights the importance of shared cultural and linguistic contexts. These insights offer valuable guidance for developing more equitable and effective multilingual language models, particularly for underrepresented languages.

Figures (3)

• Figure 1: Overview of the Analytical Process to Determine Feature Importance on LLM Performance: Starting with datasets (SIB-200 for classification and FLORES-200 for generation), we applied various multilingual language models (LLMs) and evaluated their performance. Using regression models, we predicted LLM performance in different languages based on model and language features, selected the best-performing regression model, and analyzed it with SHAP values to identify feature importance.
• Figure 2: SHAP values for Zero-shot and Two-shot Classification tasks across different models.
• Figure 3: SHAP values for Zero-shot and Two-shot Generation tasks across different models.