Benchmarking Concept-Spilling Across Languages in LLMs

TL;DR

The paper defines language spilling as cross-language semantic interference in multilingual LLMs and proposes a principled benchmark to quantify semantic robustness across nine languages using 100 high-polysemy English probes. It introduces a four-stage methodology—word selection and translation, structured meaning generation in target languages, dictionary-based judge validation, and spilling-rate computation—tied together with a judge-model concordance analysis that yields high agreement ($W=0.9176$) and strong human alignment ($\approx$ $77.43\%$). Empirically, semantic robustness varies substantially across models and languages (e.g., $41\%$ vs $20\%$ spilling on different models; German generally shows lower spilling while Spanish can peak near $47\%$), enabling principled rankings without attributing errors to a single source. The work provides a scalable benchmark and rigorous validation pipeline to guide the development of more linguistically balanced AI systems, and it commits to releasing code, datasets, and dictionaries to support reproducibility.

Abstract

Multilingual Large Language Models (LLMs) exhibit remarkable cross-lingual abilities, yet often exhibit a systematic bias toward the representations from other languages, resulting in semantic interference when generating content in non-English languages$-$a phenomenon we define as language spilling. This paper presents a novel comparative framework for evaluating multilingual semantic robustness by systematically measuring how models handle polysemous words across languages. Our methodology provides a relative measure of model performance: when required to generate exactly five meanings, both strong and weak models may resort to meanings from dominant languages, but semantically stronger models do so later in the generation sequence, producing more true meanings from the target language before failing, while weaker models resort to dominant-language meanings earlier in the sequence. We evaluate a diverse set of open and closed multilingual LLMs using a structured meaning generation task across nine languages, employing a carefully curated benchmark of 100 high-polysemy English words. Our findings reveal significant variation in semantic robustness across both models and languages, providing a principled ranking system for model comparison without requiring definitive causal attribution of error sources. We contribute both a scalable comparative benchmark for multilingual semantic evaluation and a rigorous validation pipeline$-$critical tools for developing more linguistically balanced AI systems.

Figures (6)

• Figure 1: Overview of our methodology: Starting from polysemous word selection, we generate structured meanings across languages, validate them against dictionary references, and compute spilling rates to assess semantic interference.
• Figure 2: Example of Language Spilling: The model uses the French word "cravate" (necktie) in contexts appropriate for the English word "tie" (fasten/bind), resulting in semantically unnatural sentences.
• Figure 3: Average Spilling Rate Across Generative Models. Lower is better.
• Figure 4: Average Spilling Rate Across Languages. Lower is better.
• Figure 5: Spilling Heatmap of Different Language Models Across Various Languages. Scale indicate spilling rate.