Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models

TL;DR

This work addresses the uneven multilingual performance of large language models by proposing sparse subnetwork enhancement: identify language-specific FFN neurons via Language Activation Probability Entropy (LAPE) and fine-tune only their weights for targeted languages, keeping the rest of the model frozen. The approach updates at most about 1% of parameters and outperforms full fine-tuning, FFN-only tuning, LoRA, and random-subset baselines across 12 mid- to low-resource languages on Llama-3.1-8B and Mistral-Nemo-12B, while preserving general capabilities and improving cross-lingual alignment. It demonstrates that language-specific subnetworks concentrate changes in later FFN projections, lead to favorable training dynamics, and yield stronger cross-lingual representations. In addition, the authors release language-specific neuron identifications for over 100 languages and provide an adaptation pipeline to lower the cost of extending state-of-the-art models to underrepresented languages, contributing a practical pathway toward democratizing multilingual LLM capabilities.

Abstract

Large language models exhibit uneven performance across languages, with substantial gaps between high- and low-resource languages. We present a framework for enhancing monolingual capabilities of LLMs in underrepresented languages while preserving their general-purpose performance through targeted fine-tuning of language-specific subnetworks. Our approach identifies language-specific neurons using Language Activation Probability Entropy and fine-tunes only the weights associated with these neurons, a dedicated subnetwork, on target-language data. Experiments on Llama-3.1-8B and Mistral-Nemo-12B across 12 mid- and low-resource languages demonstrate that our method consistently outperforms full fine-tuning, FFN-only fine-tuning, LoRA adaptation, and random subset fine-tuning baselines while efficiently updating only up to 1% of model parameters. Beyond performance improvements, we observe enhanced favorable training dynamics, cross-lingual representational alignment, and systematic weight update changes. To facilitate future research, we release language-specific neuron identifications for over 100 languages as well as our adaptation pipeline, offering a cost-effective pathway for adapting state-of-the-art models to underrepresented languages.

Figures (15)

• Figure 1: Sparse subnetwork adaptation method: attention blocks remain frozen (grey) while selected language-specific FFN neurons and their associated weights are fine-tuned (red). This approach enables parameter-efficient adaptation for underrepresented languages while preserving the model's general knowledge.
• Figure 2: Neuron distributions over the layers of Llama-3.1-8B for all 12 languages. The distributions for Mistral-Nemo-12B are in Appendix \ref{['app:neuron_overlap']}.
• Figure 3: Average improvement scores across all tasks for the target fine-tuning for each of the 12 languages.
• Figure 4: Validation loss curves for different fine-tuning configurations of Llama-3.1-8B on Afrikaans (all trained on 100M tokens). The plots for the other tested languages are in Appendix \ref{['app:val_losses']}.
• Figure 5: Overall improvement scores against various data sizes for a subset of 5 languages when sparsely fine-tuning Llama-3.1-8B.