The Impact of Model Scaling on Seen and Unseen Language Performance

TL;DR

The paper tackles multilingual scaling by evaluating three decoder-only LLM families (XGLM, BLOOM, BLOOMZ) across sizes on 204 languages for both text classification (SIB-200) and translation (Flores-200). Using zero-shot and $k=2$ in-context demonstrations, it reveals that scaling effects are task- and prompt-dependent: classification benefits linearly from scale in 2-shot scenarios, while translation mainly improves with instruction-tuning rather than sheer size; zero-shot performance tends to be flat with respect to model scale. A key finding is that general resource level correlates more with performance than language-specific pretraining data, though seen languages consistently outperform unseen ones and cross-language transfer helps unseen languages under certain conditions. These insights inform multilingual model design, data curation, and the trade-offs between scaling, instruction tuning, and dataset composition for broad language coverage.

Abstract

The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.

Figures (9)

• Figure 1: Multilingual performance under 0-shot and 2-shot settings. Figure \ref{['fig:0_vs_2_class']} shows results for text classification using SIB-200 topic classification dataset whereas Figure\ref{['fig:0_vs_2_gen']} shows results for text generation using Flores-200 machine translation dataset. The x-axis shows the model sizes (number of parameters in billions). The y-axis shows F1 scores (0-1) for classification task and SacreBLEU (0-100) for generation task. 'seen' indicates the languages that were present in the pretraining data mix of the models whereas 'unseen' indicates languages that were not seen by the models during pretraining.
• Figure 2: Performance of three models ( xglm; bloom; bloomz) on English-only subset of SIB-200. On the x-axis are the model sizes.
• Figure 3: bloomz's performance on seen and unseen languages for text generation task.
• Figure 4: The top row shows performance comparison between seen and unseen for languages on the classification task from the same language family across different model sizes (x-axis) for the 0-shot setting. Similarly, the bottom row shows the comparison for the generation task. & - seen; & - unseen.
• Figure 5: Classification task: Results of evaluation (F1 score) across different models based on language resource level using SIB-200 dataset. The model sizes on the x-axis are in billions of parameters. Resource Level 0, Resource Level 1, Resource Level 2, Resource Level 3, Resource Level 4, Resource Level 5.