MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining

TL;DR

MuRating introduces a scalable multilingual data-quality framework that aggregates multiple English raters via a Bradley-Terry pairwise model to produce a unified quality signal. This signal is projected to 17 target languages through translation and augmented with cross-lingual and parallel pair constraints to train a single multilingual MuRater, optimizing data selection for pretraining. Empirical results show consistent improvements over strong baselines on both English and multilingual benchmarks, with notable gains on reasoning and knowledge-intensive tasks, and across 1.2B and 7B parameter LLaMA-style models. The work highlights translation fidelity and cross-lingual regularization as critical factors for effective multilingual data curation at scale.

Abstract

Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a single rater for 17 target languages. MuRating aggregates multiple English "raters" via pairwise comparisons to learn unified document-quality scores,then projects these judgments through translation to train a multilingual evaluator on monolingual, cross-lingual, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain a 1.2 B-parameter LLaMA model. Compared to strong baselines, including QuRater, AskLLM, DCLM and so on, our approach boosts average accuracy on both English benchmarks and multilingual evaluations, with especially large gains on knowledge-intensive tasks. We further analyze translation fidelity, selection biases, and underrepresentation of narrative material, outlining directions for future work.

Figures (16)

• Figure 1: Overview of the MuRating pipeline: English document pairs are first annotated using various data selection methods and unified, then translated into multiple languages to create diverse multilingual pairs. These are used to train the MuRater model, which scores large-scale web data. The top 10% of scored data is selected to train an LLM, yielding superior performance compared to state-of-the-art sampling baselines.
• Figure 2: Performance of different selection methods on ARC-Challenge-ML, MMLU-ML, XWinograd, and the overall average across all tasks during training on 200B English + 300B multilingual tokens.
• Figure 3: Performance of different selection methods on ARC-Challenge, HellaSwag, TriviaQA, and the overall average across 12 tasks during training on 200B English tokens
• Figure 4: Scatter plots of scores assigned by multilingual raters to 10,000 parallel documents across various languages. Green points represent ratings from raters trained with alignment using parallel and cross-lingual pairs, while blue points indicate scores from unaligned raters.
• Figure 5: Scatter plots of average scores assigned by GPT-4o to Arabic and Spanish parallel data. Each point represents an average of 20 evaluations. Left: pointwise scoring. Right: pairwise scoring.