Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts
Guorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen, Yuping Zheng, Benyou Wang
TL;DR
This work targets the data scarcity and scalability barriers of medical LLMs across 50 languages. It introduces a language-aware Mixture of Experts framework with Hybrid routing and a circuits-based interpretation to reveal how cross-lingual information is shared in early layers and language-specific processing emerges later (Spread Out in the End). Building on Post-MoE, routing is concentrated in the final layers to balance general medical knowledge with language specialization, and language-family experts (Apollo-MoE) scale the approach efficiently to 50 languages without increasing parameters. Empirical results show improved multilingual generalization, interpretability, and data-efficient scaling across 0.5B–7B base models, with Apollo-MoE achieving strong performance in both high- and low-resource languages. Overall, the method offers a practical, scalable path to democratize medical knowledge through multilingual LLMs while preserving transferability and interpretability.
Abstract
Adapting medical Large Language Models to local languages can reduce barriers to accessing healthcare services, but data scarcity remains a significant challenge, particularly for low-resource languages. To address this, we first construct a high-quality medical dataset and conduct analysis to ensure its quality. In order to leverage the generalization capability of multilingual LLMs to efficiently scale to more resource-constrained languages, we explore the internal information flow of LLMs from a multilingual perspective using Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE routing method that employs language-specific experts and cross-lingual routing. Inspired by circuit theory, our routing analysis revealed a Spread Out in the End information flow mechanism: while earlier layers concentrate cross-lingual information flow, the later layers exhibit language-specific divergence. This insight directly led to the development of the Post-MoE architecture, which applies sparse routing only in the later layers while maintaining dense others. Experimental results demonstrate that this approach enhances the generalization of multilingual models to other languages while preserving interpretability. Finally, to efficiently scale the model to 50 languages, we introduce the concept of language family experts, drawing on linguistic priors, which enables scaling the number of languages without adding additional parameters.