Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering

TL;DR

This work investigates multilingual processing in Mixture-of-Experts architectures to uncover how sparse routing and expert specialization drive cross-language transfer. Through systematic analyses of routing distributions, language-associated experts, and layerwise interventions on a representative MoE system evaluated on Belebele, MGSM, and FLORES-200, the study reveals that routing aligns with linguistic families and that middle layers act as language-agnostic capacity hubs while early and late layers preserve language-specific processing. A routing-guided steering method is proposed to bias middle-layer routing toward shared experts tied to dominant languages, yielding consistent multilingual gains, especially for linguistically related pairs, and demonstrating robust cross-lingual capacity sharing without disrupting language-specific understanding. The findings offer an architecture-aware perspective on multilingual enhancement and motivate inference-time interventions that exploit structured routing to improve cross-language performance. The work also discusses limitations and future directions, including scaling to larger MoE setups and exploring training-time or post-training interventions to shape expert behavior. Overall, the results provide mechanistic insight into how sparsity and layerwise organization enable cross-lingual transfer in MoE LLMs and offer practical steering techniques to harness shared capacity for multilingual robustness.

Abstract

Mixture-of-Experts (MoE) architectures have shown strong multilingual capabilities, yet the internal mechanisms underlying performance gains and cross-language differences remain insufficiently understood. In this work, we conduct a systematic analysis of MoE models, examining routing behavior and expert specialization across languages and network depth. Our analysis reveals that multilingual processing in MoE models is highly structured: routing aligns with linguistic families, expert utilization follows a clear layerwise pattern, and high-resource languages rely on shared experts while low-resource languages depend more on language-exclusive experts despite weaker performance. Layerwise interventions further show that early and late MoE layers support language-specific processing, whereas middle layers serve as language-agnostic capacity hubs. Building on these insights, we propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time, leading to consistent multilingual performance improvements, particularly for linguistically related language pairs. Our code is available at https://github.com/conctsai/Multilingualism-in-Mixture-of-Experts-LLMs.

Figures (9)

• Figure 1: Pairwise cross-language routing similarity across different languages. Red indicates high similarity, while blue indicates high dissimilarity.
• Figure 2: Layer-wise cross-language routing similarity. The figure comprises three subplots representing pairwise combinations of three representative languages: English (En), Chinese (Zh) and Arabic (Ar).
• Figure 3: Average number of exclusive experts per layer for different language groups.
• Figure 4: Case study of layerwise intervention on Chinese-exclusive experts. Early-layer intervention leads to understanding failure, while late-layer intervention results in language mixing.
• Figure 5: Routing entropy per layer for each language.