Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation

TL;DR

The paper investigates why multilingual language models underperform monolinguals on lexical disambiguation by leveraging controlled datasets in English and Spanish. It tests three capacity-related hypotheses—representational isotropy, attentional allocation, and vocabulary/tokenization—and quantifies them using Center-Isotropy ($CI$), attention analyses, and token counts. Using regression analyses of disambiguation performance measured by $R^2$, Center-Isotropy ($CI$), and model fit via $AIC$, the authors show that multilingual status predicts lower performance beyond size and depth, with isotropy and tokenization as the strongest correlates and a counterintuitive link for attention. These findings provide a proof-of-concept that multiple capacity bottlenecks drive the multilingual penalty and point to concrete directions for mitigating it.

Abstract

Multilingual language models (LMs) sometimes under-perform their monolingual counterparts, possibly due to capacity limitations. We quantify this ``multilingual penalty'' for lexical disambiguation--a task requiring precise semantic representations and contextualization mechanisms--using controlled datasets of human relatedness judgments for ambiguous words in both English and Spanish. Comparing monolingual and multilingual LMs from the same families, we find consistently reduced performance in multilingual LMs. We then explore three potential capacity constraints: representational (reduced embedding isotropy), attentional (reduced attention to disambiguating cues), and vocabulary-related (increased multi-token segmentation). Multilingual LMs show some evidence of all three limitations; moreover, these factors statistically account for the variance formerly attributed to a model's multilingual status. These findings suggest both that multilingual LMs do suffer from multiple capacity constraints, and that these constraints correlate with reduced disambiguation performance.

Figures (5)

• Figure 1: The best-performing layers of multilingual models generally exhibited reduced performance (as measured by $R^2$) compared to monolingual models of equivalent size.
• Figure 2: Compared to their monolingual counterparts, multilingual models models showed evidence of reduced isotropy (left) and somewhat reduced attention to disambiguating cues (right).
• Figure 3: The $AIC$ (scaled to the Baseline model) associated with linear mixed models predicting disambiguation performance from various factors (lower is better).
• Figure 4: Multilingual models showed evidence of reduced isotropy (lower average cosine distance between token embeddings; and higher cosine similarity between individual token embeddings and the sentence average) relative to monolingual models.
• Figure 5: Multilingual models consistently segmented target words into more tokens than monolingual models.