Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

TL;DR

Aleph-Alpha-GermanWeb addresses data scarcity for German LLM pre-training by integrating a model-based data curation pipeline with synthetic data generation. The approach combines Common Crawl data (filtered and cleaned), FineWeb2, and synthetic data conditioned on organic sources, and evaluates on both 1B Llama-style and 8B HAT models. Across benchmarks such as MMMLU, ARC, HellaSwag, and TruthfulQA, the GermanWeb datasets consistently surpass FineWeb2, even when FineWeb2 is augmented with high-quality sources like Wikipedia. The work demonstrates that data quality and synthetic augmentation can yield substantial gains, and it publicly releases the dataset to spur further German-language LLM development, while noting careful considerations around translation quality and potential synthetic-data-induced model dynamics.

Abstract

Scaling data quantity is essential for large language models (LLMs), yet recent findings show that data quality can significantly boost performance and training efficiency. We introduce a German-language dataset curation pipeline that combines heuristic and model-based filtering techniques with synthetic data generation. We use our pipeline to create Aleph-Alpha-GermanWeb, a large-scale German pre-training dataset which draws from: (1) Common Crawl web data, (2) FineWeb2, and (3) synthetically-generated data conditioned on actual, organic web data. We evaluate our dataset by pre-training both a 1B Llama-style model and an 8B tokenizer-free hierarchical autoregressive transformer (HAT). A comparison on German-language benchmarks, including MMMLU, shows significant performance gains of Aleph-Alpha-GermanWeb over FineWeb2 alone. This advantage holds at the 8B scale even when FineWeb2 is enriched by human-curated high-quality data sources such as Wikipedia. Our findings support the growing body of evidence that model-based data curation and synthetic data generation can significantly enhance LLM pre-training datasets.

Figures (3)

• Figure 1: Average accuracies (MMMLU, ARC-Easy, HellaSwag; single- & five-shot) of 1B Llama-style models trained on $\sim$ 84 billion tokens from different datasets. All subsets of Aleph-Alpha-GermanWeb -- marked with an asterisk (*) -- outperform FineWeb2. For comparison, we also include EPFML High. Here, 1,000 training steps equates to approximately 2.1 billion tokens. Individual benchmark results are provided in App. \ref{['app:1B-benchmarks']}.
• Figure 2: Accuracies of 8B HAT models trained on different datasets. On average, and for all but one individual benchmark comparison, Aleph-Alpha-GermanWeb outperforms FineWeb2. (Left) The HAT model was trained for 25,000 training steps on an English-language web-derived dataset, equating to $\sim$ 75 billion English words. Afterwards the model was further trained for 20,000 steps ($\sim$ 60 billion words) with German language data, either random FineWeb2 data augmented with high quality datasets like Wikipedia (FineWeb2:HQ Curated) or our generated synthetic data (Synthetic). Both datasets amount to 60 billion German words each. (Middle) We trained for 21,000 steps ($\sim$ 63 billion words) on either random FineWeb2 data or on our synthetic German data. (Right) The FineWeb2 training run was continued to 50,0000 steps and is compared to a training of 50,000 steps on data from our Filtered CC pipeline. Both runs each equate to $\sim$ 150 billion words.
• Figure 3: Individual benchmark results for 1B Llama-style models trained on $\sim$ 84 billion tokens from different datasets. Here, 1,000 training steps equates to approximately 2.1 billion tokens.