The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?

TL;DR

This survey investigates why multilingual language models exhibit uneven performance across languages, arguing that many gaps arise from design choices—tokenization, encoding, data exposure, and parameter sharing—rather than intrinsic linguistic difficulty. By mapping linguistic properties to concrete modeling mechanisms and design levers, the authors show that normalizing segmentation, encoding, and exposure substantially reduces cross-linguistic disparities. They synthesize actionable recommendations across tokenization, sampling, architectures, and evaluation, advocating typology-aware benchmarks and language-adaptive pretraining to promote balanced performance. The work highlights practical implications for building more equitable multilingual NLP systems and outlines future directions in curriculum design, capacity modeling, and language-aware evaluation. The significance lies in reframing linguistic diversity as a design constraint that can be addressed with targeted tooling and evaluation strategies, enabling broader and fairer access to multilingual NLP capabilities.

Abstract

Multilingual language models (LMs) promise broader NLP access, yet current systems deliver uneven performance across the world's languages. This survey examines why these gaps persist and whether they reflect intrinsic linguistic difficulty or modeling artifacts. We organize the literature around two questions: do linguistic disparities arise from representation and allocation choices (e.g., tokenization, encoding, data exposure, parameter sharing) rather than inherent complexity; and which design choices mitigate inequities across typologically diverse languages. We review linguistic features, such as orthography, morphology, lexical diversity, syntax, information density, and typological distance, linking each to concrete modeling mechanisms. Gaps often shrink when segmentation, encoding, and data exposure are normalized, suggesting much apparent difficulty stems from current modeling choices. We synthesize these insights into design recommendations for tokenization, sampling, architectures, and evaluation to support more balanced multilingual LMs.