A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce

TL;DR

<3-5 sentence high-level summary> The paper investigates RL-based post-training of LLMs for mathematical reasoning, showing that a simple rejection-sampling baseline (RAFT) and its enhanced variant RAFT++ can approach the performance of more complex methods like GRPO and PPO. A key finding is that GRPO's advantage largely stems from discarding prompts with entirely incorrect responses, rather than reward normalization. Building on this, the authors propose Reinforce-Rej, a minimal policy-gradient extension that filters both entirely incorrect and entirely correct samples, achieving comparable final performance with improved KL efficiency. The work argues for treating negative signals more selectively and highlights RAFT and Reinforce-Rej as robust, interpretable baselines for reward-based LLM post-training, with practical implications for future method design in this space.

Abstract

Reinforcement learning (RL) has become a prevailing approach for fine-tuning large language models (LLMs) on complex reasoning tasks. Among recent methods, GRPO stands out for its empirical success in training models such as DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In this work, we revisit GRPO from a reinforce-like algorithm perspective and analyze its core components. Surprisingly, we find that a simple rejection sampling baseline, RAFT, which trains only on positively rewarded samples, yields competitive performance than GRPO and PPO. Our ablation studies reveal that GRPO's main advantage arises from discarding prompts with entirely incorrect responses, rather than from its reward normalization. Motivated by this insight, we propose Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples. Reinforce-Rej improves KL efficiency and stability, serving as a lightweight yet effective alternative to more complex RL algorithms. We advocate RAFT as a robust and interpretable baseline, and suggest that future advances should focus on more principled designs for incorporating negative samples, rather than relying on them indiscriminately. Our findings provide guidance for future work in reward-based LLM post-training.

Figures (4)

• Figure 1: The learning dynamics of RAFT and RAFT++, initialized from Qwen2.5-Math-7B-base (left) and LLaMA-3.2-3B-instruct (right). The y-axis is the average@16 accuracy, that is further averaged on MATH500, Minerva Math, and Olympiad Bench. We also plot the best model of GRPO, PPO, and Iterative DPO for reference.
• Figure 2: Left: the training reward curves of RAFT, RAFT++, RAFT++ without clipping (i.e., RAFT and importance sampling), and GRPO, initialized from Qwen2.5-Math-7B-base. Right: the training reward curves of RAFT++ and RAFT++ enhanced by clip higher trick, initialized from LLaMA-3.2-3B-instruct. We transform the original reward using $(1 + r)/2$ so that the resulting value corresponds to the accuracy on the training data. We also apply a moving average with a window size of $20$ to smooth the curves.
• Figure 3: The learning dynamics of RAFT++ and GRPO, initialized from Qwen2.5-Math-7B-base (first row) and LLaMA-3.2-3B-instruct (second row). We also plot the KL loss in the left column and the policy entropy loss in the right column.
• Figure 4: Ablation study on the components of GRPO and Reinforce-type algorithms with LLaMA-3.2-3B-instruct. We compare GRPO with other Reinforce-based variants to isolate the effects of removing incorrect samples, correct samples, and applying normalization. Removing incorrect samples (“Remove all wrong”) provides the largest gain in reward, highlighting their harmful impact. In contrast, the reward of removing correct samples is still not satisfactory. Mean-zero normalization increases KL loss and destabilizes training. Normalizing by standard deviation shows minimal additional benefit. The variant “Reinforce + Remove both” achieves a good balance between reward, KL stability, and entropy regularization. We transform the original reward using $(1 + r)/2$ so that the resulting value corresponds to the accuracy on the training data. We also apply a moving average with a window size of $20$ to smooth the curves.