Gaperon: A Peppered English-French Generative Language Model Suite

TL;DR

Gaperon presents a fully open suite of French–English LLMs at 1.5B, 8B, and 24B parameters trained on 2–4T tokens, with an end-to-end data pipeline including neural quality filtering and near-deduplication. The work investigates how data curation and contamination influence generation quality and benchmark performance, introducing a family of variants (Young, Pepper, Garlic) and aPenicillin-based contamination framework to probe safety and evaluation dynamics. Key findings include that linguistic-quality filtering improves fluency but can depress benchmark scores, while late deliberate contamination can recover competitive benchmarks at the cost of generation quality; moreover, high-quality filtering can unintentionally amplify leakage in benchmarks. By releasing datasets, code, and hundreds of intermediate checkpoints, Gaperon provides a transparent testbed for studying data-curation trade-offs, multilingual safety, and benchmarking integrity in large-scale open LLM development.

Abstract

We release Gaperon, a fully open suite of French-English-coding language models designed to advance transparency and reproducibility in large-scale model training. The Gaperon family includes 1.5B, 8B, and 24B parameter models trained on 2-4 trillion tokens, released with all elements of the training pipeline: French and English datasets filtered with a neural quality classifier, an efficient data curation and training framework, and hundreds of intermediate checkpoints. Through this work, we study how data filtering and contamination interact to shape both benchmark and generative performance. We find that filtering for linguistic quality enhances text fluency and coherence but yields subpar benchmark results, and that late deliberate contamination -- continuing training on data mixes that include test sets -- recovers competitive scores while only reasonably harming generation quality. We discuss how usual neural filtering can unintentionally amplify benchmark leakage. To support further research, we also introduce harmless data poisoning during pretraining, providing a realistic testbed for safety studies. By openly releasing all models, datasets, code, and checkpoints, Gaperon establishes a reproducible foundation for exploring the trade-offs between data curation, evaluation, safety, and openness in multilingual language model development.

Figures (16)

• Figure 1: Pretraining data quality experiments. Scores are the average of the following English tasks: ARC-Easy allenai:arc, Arc-Challenge allenai:arc, Hellaswag zellers2019hellaswag, SciQ SciQ and PIQA Bisk2020piqa.
• Figure 2: Evaluation of the convergence of our Scaled RMS Norm approach in the True precision setup ($C=50$). We minimize the mean squared coefficients of the output of an RMS layer fed with random gaussian inputs (dimension 32, batch size 12, 1000 optimization steps). We observe that our Scaled RMS Norm converges for a wider range of learning rates than the Vanilla RMS Norm in bfloat16 precision.
• Figure 3: Performance comparison between Headless and Vanilla models across training duration, showing average scores on French and English benchmarks for both 1B and 8B model sizes. Headless models (blue) achieve faster training but show performance stagnation, while Vanilla models (orange) continue improving with extended training. For the 1B models, English benchmarks include ARC-E, ARC-C, HellaSwag, LAMBADA, SciQ, and PIQA; French benchmarks include ARC-C and HellaSwag. For the 8B models, benchmarks include additionally BoolQ for English, and LAMBADA for French.
• Figure 4: Summary of the Gaperon-1.5B training run. Using the average scores from: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, ARC-C-Fr, Hellaswag-Fr (5-shot).
• Figure 5: Summary of the Gaperon-8B training run. Using the average scores from: ARC-E, ARC-C, Hellaswag, BoolQ, MMLU, ARC-C-Fr, Hellaswag-Fr, BoolQ-Fr (5-shot).