Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers

TL;DR

Babel presents an open multilingual LLM that covers 25 top-speaking languages to reach over 90% of the world’s population. It achieves this by expanding model capacity through a layer-extension approach rather than traditional continual pretraining, and by constructing two variants, Babel-9B and Babel-83B, optimized for inference, fine-tuning, and multilingual performance. Through a diverse data pipeline, LLM-based quality control, and staged pre-training (Recovery then Continuous Training), Babel demonstrates state-of-the-art open multilingual performance on a broad set of benchmarks and strong results in supervised fine-tuning, including near-commercial capabilities on some tasks. The work highlights the viability and impact of broad-language open models for multilingual understanding, translation, and reasoning, and it provides practical baselines and datasets for further research in low-resource language coverage.

Abstract

Large language models (LLMs) have revolutionized natural language processing (NLP), yet open-source multilingual LLMs remain scarce, with existing models often limited in language coverage. Such models typically prioritize well-resourced languages, while widely spoken but under-resourced languages are often overlooked. To address this disparity, we introduce $\texttt{Babel}$, an open multilingual LLM that covers the top 25 languages by number of speakers, supports over 90% of the global population, and includes many languages neglected by other open multilingual LLMs. Unlike traditional continue pretraining approaches, Babel expands its parameter count through a layer extension technique that elevates Babel's performance ceiling. We introduce two variants: $\texttt{Babel-9B}$, designed for efficient inference and fine-tuning, and $\texttt{Babel-83B}$, which sets a new standard for open multilingual LLMs. Extensive evaluations on multilingual tasks demonstrate its superior performance compared to open LLMs of comparable size. In addition, using open-source supervised fine-tuning datasets, Babel achieves remarkable performance, with Babel-9B-Chat leading among 10B-sized LLMs and Babel-83B-Chat setting a new standard for multilingual tasks, reaching the same level of commercial models.

Figures (4)

• Figure 1: Layer extension for Babel.
• Figure 2: Performance of Babel-9B-Base comparison across languages.
• Figure 3: Multilingual SFT data distribution excluding English and Chinese.
• Figure 6: Performance comparison of English and multilingual SFT data.