The Impact of Post-training on Data Contamination

TL;DR

This study evaluates how data contamination interacts with post-training in large language models by injecting contamination into a $25\text{B}$-token pre-training corpus and examining outcomes after extended pre-training and after two post-training paradigms, SFT and GRPO. Using Qwen2.5 ($0.5/1.5$B) and Gemma3 ($1/4$B) variants, the authors find that contamination creates a transient performance spike during exposure, which fades with more pre-training but re-emerges under post-training. SFT tends to inflate scores on contaminated benchmarks only, whereas GRPO inflates performance on both contaminated and uncontaminated benchmarks, indicating different implications for memorization versus generalization. Model scale amplifies these effects: larger models exhibit stronger memorization under SFT but broader generalization under GRPO, highlighting the need for contamination audits after post-training and suggesting RL-based post-training can mitigate some over-estimation risks. These findings inform evaluation practices and mitigation strategies across the full model life cycle, from pre-training through alignment and deployment.

Abstract

We present a controlled study of how dataset contamination interacts with the post-training stages now standard in large language model training pipelines. Starting from clean checkpoints of Qwen2.5 (0.5B/1.5B) and Gemma3 (1B/4B), we inject five copies of GSM8K and MBPP test items into the first 2B tokens of an otherwise 25B token extended pre-training dataset. We then compare the contaminated and clean models both immediately after pre-training and again after two popular post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL) with group relative policy optimization (GRPO). The applied post-training steps do not have any contamination. Across math and coding benchmarks, we find three consistent patterns: (i) Contamination causes performance spikes that are gradually diminished with continued pre-training. After even 25B tokens the apparent performance inflation of contamination can become close to zero. (ii) Both SFT and GRPO resurface the leaked information, but with different external validity: SFT inflates scores only on the contaminated tasks, whereas GRPO also inflates performance on uncontaminated counterparts (GSMPlus, HumanEval). (iii) Model scale amplifies these tendencies, larger Supervised Fine Tuned models memorize more, while larger GRPO models translate leakage into more generalizable capabilities. Our results underscore the need for contamination audits \emph{after} post-training and suggest that RL-based post-training, although not immune, can help alleviate contamination-related over-estimation problems.

Figures (6)

• Figure 1: An Overview of our Method: We take existing pre-trained models and run them through extended pre-training with and without contamination. Afterwards we post-train them using SFT or RL methods and compare their performance. The pre-trained checkpoints here are from Qwen2.5 and Gemma3 non-instruction tuned models.
• Figure 2: Performance Over Time: Accuracy and Pass@1 of the Clean and Contaminated Qwen2.5-1.5B models on the GSM8K Benchmark. The contamination is within the first 500 steps. We observe that the performance gap is much bigger at the exposure point but then closes as the model is trained on more data. For MBPP we observe that the peak is much higher meaning contamination of code is memorized better at sight however overtime the performance normalizes just like math.
• Figure 3: Performance Difference on Math: Accuracy difference between Contaminated and Clean models right after pre-training (base) and after the SFT and GRPO steps on the GSM8K benchmark. We observe that while the Base differences show little to no impact from contamination post-training can actually uncover the information acquired by the model in pre-training even after additional training seem to have covered it.
• Figure 4: Performance Difference on Code: Accuracy difference between Contaminated and Clean models right after pre-training(base) and after the SFT and GRPO steps on the MBPP benchmark. The same trend with Figure \ref{['fig:math_diff']} holds for the code benchmarks as well. Here we only present the larger models as the smaller non-instruction tuned models had noisy evals for the coding benchmarks.
• Figure 5: Comparison of Performance gap on contaminated and uncontaminated datasets. We observe that the Base models behave roughly the same on the contaminated and uncontaminated datasets for both Math and Coding. GRPO fine-tuned models have a positive gap on the contaminated dataset but also have a smaller but still positive gap on the uncontaminated dataset meaning suggesting the models learn some generalizable patterns. The SFT models on the other hand only have a larger gap in the contaminated dataset and show almost no improvement on the contaminated data.