Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension

Rongzhi Li, Hitomi Yanaka

TL;DR

It is found that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers, which reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.

Abstract

Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose $\textbf{NeuronMoE}$, a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.

NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension

TL;DR

It is found that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers, which reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.

Abstract

Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose , a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.
Paper Structure (39 sections, 6 equations, 3 figures, 11 tables)

This paper contains 39 sections, 6 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Extension via Mixture-of-Experts
  4. Language-Specific Neurons in Multilingual Models
  5. Identification Methods:
  6. Layer-wise Heterogeneity:
  7. Current Applications and Limitations:
  8. Bridging Neuron Analysis and MoE Allocation:
  9. Method
  10. Neuron Specialization Measurement
  11. Language-Specific Neuron Definition:
  12. Cross-Lingual Neuron Diversity for Expert Allocation:
  13. Expert Allocation Strategy
  14. Two-Stage Training Process
  15. Stage 1 (Expert Initialization):
  16. ...and 24 more sections

Figures (3)

  • Figure 1: Expert allocation comparison on Llama-3.2-3B for Greek extension. Cross-lingual neuron diversity (red line) exhibits heterogeneous distribution across layers. NeuronMoE allocates experts following this distribution, achieving 41.7% parameter reduction (49 vs 84 experts) compared to LayerMoE's uniform allocation strategy.
  • Figure 2: Overview of NeuronMoE's core insight and allocation strategy. Top: Cross-lingual neuron diversity exhibits dramatic heterogeneity across layers. For example, Layer 0 contains 342 unique language-specific neurons while Layer 10 contains only 7, demonstrating that different layers have vastly different language-specific processing requirements. Bottom: NeuronMoE allocates experts following this empirical distribution (49 experts total), concentrating capacity where specialization occurs. In contrast, LayerMoE's similarity-based approach over-allocates in middle layers (84 experts total). This neuron-guided strategy achieves 41.7% parameter reduction while maintaining comparable performance.
  • Figure 3: Heatmap of high-AP neuron ratios for Greek across all experts. Color intensity indicates Greek specialization strength (red: high, blue: low/none). Multi-expert layers (0, 21, 23-27) show distinct specialization patterns across experts, while single-expert middle layers remain largely unspecialized, validating our neuron-guided allocation strategy.