The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining

TL;DR

The paper examines how mixed-language documents shape multilingual LLM pretraining by constructing a controlled MonoWeb corpus that removes multilingual documents and comparing it to FineWeb. It introduces a two-stage bilingual data identification pipeline and a granular ablation that reveals parallel data as the dominant driver of translation performance, responsible for restoring most of the BLEU score when reintroduced, while code-switching data contributes minimally. Across cross-lingual QA and reasoning, bilingual data have a substantially smaller impact, with performance largely preserved even in mono-lingual pretraining. The findings highlight a clear task-specific reliance on cross-lingual signals, suggesting prioritizing high-quality parallel data for translation and leveraging monolingual exposure for broader cross-lingual understanding, with implications for data collection and pretraining objectives.

Abstract

Multilingual large language models achieve impressive cross-lingual performance despite largely monolingual pretraining. While bilingual data in pretraining corpora is widely believed to enable these abilities, details of its contributions remain unclear. We investigate this question by pretraining models from scratch under controlled conditions, comparing the standard web corpus with a monolingual-only version that removes all multilingual documents. Despite constituting only 2% of the corpus, removing bilingual data causes translation performance to drop 56% in BLEU, while behaviour on cross-lingual QA and general reasoning tasks remains stable, with training curves largely overlapping the baseline. To understand this asymmetry, we categorize bilingual data into parallel (14%), code-switching (72%), and miscellaneous documents (14%) based on the semantic relevance of content in different languages. We then conduct granular ablations by reintroducing parallel or code-switching data into the monolingual-only corpus. Our experiments reveal that parallel data almost fully restores translation performance (91% of the unfiltered baseline), whereas code-switching contributes minimally. Other cross-lingual tasks remain largely unaffected by either type. These findings reveal that translation critically depends on systematic token-level alignments from parallel data, whereas cross-lingual understanding and reasoning appear to be achievable even without bilingual data.

Figures (5)

• Figure 1: Performance on WMT14 for different pretraining setups. FineWeb: multilingual web data from FineWeb and FineWeb-2; MonoWeb: multilingual web data with bilingual documents removed.
• Figure 2:
• Figure 3:
• Figure 4:
• Figure 5: