AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

TL;DR

AfriqueLLM systematically analyzes how CPT data composition and base-model architecture affect adaptation to 20 African languages. By training on a 26B-token mixed corpus across five backbones (Llama 3.1, Gemma 3, Qwen 3) and varying data mixtures (monolingual, code, math, synthetic, translation), the study shows data composition as the primary driver of gains, with CMS-style mixtures delivering strong improvements while preserving reasoning. A key finding is the 'Zero-to-Hero' effect observed with Qwen 3 when CPTed, where strong HRL priors enable substantial performance boosts on Africa-centric tasks, including document-level translation, often surpassing larger but less capable baselines. The work provides practical guidance on data selection and model choice for African-language adaptation and releases the AfriqueLLM checkpoints to advance region-specific NLP research.

Abstract

Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present \texttt{AfriqueLLM}, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models have been released on [Huggingface](https://huggingface.co/collections/McGill-NLP/afriquellm).

Figures (1)

• Figure 1: Performance comparison across language groups: High-Resource PT (English), African PT, and African Non-PT. We report the average score across all benchmarks excluding Flores. Hatched bars represent base models, while solid bars represent their Afrique-adapted counterparts. Values above the bars indicate the absolute improvement ($\Delta$) after adaptation.