ReDit: Reward Dithering for Improved LLM Policy Optimization

TL;DR

The paper tackles gradient instability in LLM policy optimization caused by discrete rewards by introducing Reward Dithering (ReDit), which adds zero-mean noise to rewards to create informative gradient signals without bias. ReDit preserves the GRPO optimization structure while increasing reward variance, yielding unbiased gradient estimates and enhanced exploration. Empirically, ReDit accelerates convergence and improves final accuracy across GSM8K, MATH, Geometry3K, code benchmarks, and various baselines, with Gaussian smoothing often performing best. Theoretical results show that the induced gradient noise helps prevent vanishing/exploding gradients, providing a principled basis for the observed improvements and guiding future work on automated variance control.

Abstract

DeepSeek-R1 has successfully enhanced Large Language Model (LLM) reasoning capabilities through its rule-based reward system. While it's a ''perfect'' reward system that effectively mitigates reward hacking, such reward functions are often discrete. Our experimental observations suggest that discrete rewards can lead to gradient anomaly, unstable optimization, and slow convergence. To address this issue, we propose ReDit (Reward Dithering), a method that dithers the discrete reward signal by adding simple random noise. With this perturbed reward, exploratory gradients are continuously provided throughout the learning process, enabling smoother gradient updates and accelerating convergence. The injected noise also introduces stochasticity into flat reward regions, encouraging the model to explore novel policies and escape local optima. Experiments across diverse tasks demonstrate the effectiveness and efficiency of ReDit. On average, ReDit achieves performance comparable to vanilla GRPO with only approximately 10% the training steps, and furthermore, still exhibits a 4% performance improvement over vanilla GRPO when trained for a similar duration. Visualizations confirm significant mitigation of gradient issues with ReDit. Moreover, theoretical analyses are provided to further validate these advantages.

Figures (19)

• Figure 1: Training Dynamics of Gradient Norm and Reward for Qwen2.5-7B qwen2025qwen25technicalreport on GSM8K cobbe2021trainingverifierssolvemath Dataset. The left and right figures compare original gradient norm (before gradient clipping Zhang2020Why) and reward trends across training steps. The original GRPO method (the left figure) suffers from significant gradient instability—both vanishing (red dots, norms < 0.01) and exploding (purple asterisks, norms > 5). In contrast, ReDit with Gaussian reward smoothing (the right figure) effectively stabilizes optimization throughout training.
• Figure 2: The figure illustrates how ReDit of different variances gradually smooth the reward distribution, showing the smoothing effect of perturbations of different variances on the reward distribution.
• Figure 3: Qwen2.5-7B qwen2025qwen25technicalreport Gradient norm and reward training dynamics of standard GRPO on GSM8k and MATH datasets. During the whole optimization process, the gradient of standard GRPO is unstable, and there are a lot of gradient vanishing or gradient exploding cases.
• Figure 4: GRPO has unstable performance on the MATH test set. The figure plots the test accuracy achieved for the checkpoints saved during the training run shown in Fig. \ref{['fig:Training Dynamics GRPO']}(the right figure).
• Figure 5: Accuracy of different GRPO variants (DAPO, DR.GRPO, REINFORCE++) tested on the GSM8K dataset. The horizontal dashed line highlights the performance of using ReDit at about 1000 training steps, and even after 9000 steps, its accuracy is comparable to the baseline.

Theorems & Definitions (8)

• Theorem 1: Policy network optimization time lower bound
• Theorem 2: Policy network optimization time upper bound
• Proposition 1: Unbiased estimate of gradient
• Proposition 2: Introducing the variance of gradient estimation
• Definition 1
• Definition 2
• proof
• proof