LOLA -- An Open-Source Massively Multilingual Large Language Model

TL;DR

LOLA tackles the challenge of multilingual generalization with a compute-efficient, decoder-only Transformer that uses sparse MoE layers to scale capacity without a proportional rise in compute. Trained on CulturaX data spanning $167$ languages, it achieves competitive results across 13 multilingual benchmarks and reveals that MoE routing organizes tokens by language-family patterns, even with modest active parameters ($1.3$B) out of $7.4$B total. The study systematically compares LOLA to 17 baselines, showing strong performance against smaller models while acknowledging gaps against the largest Category-3 models, particularly in Q&A and few-shot settings. As an open-source contribution, LOLA demonstrates the potential of sparse MoE for scalable, multilingual NLP and provides insights into language-group specialization via expert routing, with implications for future research and practical deployment.

Abstract

This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages.

Figures (24)

• Figure 1: Three-level overview of the LOLA architecture. The left-most block provides a high-level overview of the layers within LOLA, including the alternating standard and MoE-based decoder blocks. The middle block gives a detailed view of the MoE-based decoder block structure. The right-most block zooms in on the inner workings of each MoE layer, showing how the top-1 gating mechanism selects from multiple expert .
• Figure 2: Comparison of LOLA's zero-, one- and few-shot performance against the other multilingual models across all supported combinations of tasks and languages, categorized by model size. The left side shows the results from the Wilcoxon signed-rank test, indicating whether LOLA significantly outperforms (Wins), shows no significant difference (Inconclusive) or is outperformed by (Losses) other models. On the right is the average performance comparison to confirm whether LOLA is on average better than (Wins), the same as (Ties), or worse than (Losses) the other models.
• Figure 3: Distortion (top) and Silhouette (bottom) score graphs for K-Means clustering with $k$ values up to 10. The clusters are based on the number of active parameters in the models.
• Figure 4: Comparison of model sizes across all evaluated models, with our model highlighted in orange. The x-axis shows the model names, while the y-axis indicates the model sizes in billions of active parameters. The models are grouped into three size categories: Category-1, Category-2, and Category-3. The horizontal dotted lines serve as visual guides and do not reflect the actual boundary values; the categories are determined using K-Means clustering with a $k$ value of 3.
• Figure 5: Pearson correlation values for distance between languages based on phylogenetic features and LOLA's MoE routing features. The x-axis represents the numbers of languages included in the comparison. We include the languages in the descending order of the number of documents seen by the model for that language.