GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping

TL;DR

This work identifies implicit reward hacking in GRPO-based diffusion-flow models caused by biased importance-ratio clipping, where the ratio $r_t(\theta)$ has a left-shifted mean and timestep-dependent variance. It introduces GRPO-Guard, combining RatioNorm to normalize the log-importance ratio and a gradient-balancing scheme with a reweighting factor $\delta$ to ensure consistent clipping and balanced updates across denoising steps. The approach stabilizes training across diffusion backbones (e.g., SD3.5-M, Flux.1-dev) and proxy tasks, reducing over-optimization while preserving or improving generation quality, and does so without heavy KL regularization. Overall, GRPO-Guard offers a robust, generalizable method for safe policy optimization in diffusion-based generation by correcting ratio-based biases and harmonizing gradient flows across timesteps.

Abstract

Recently, GRPO-based reinforcement learning has shown remarkable progress in optimizing flow-matching models, effectively improving their alignment with task-specific rewards. Within these frameworks, the policy update relies on importance-ratio clipping to constrain overconfident positive and negative gradients. However, in practice, we observe a systematic shift in the importance-ratio distribution-its mean falls below 1 and its variance differs substantially across timesteps. This left-shifted and inconsistent distribution prevents positive-advantage samples from entering the clipped region, causing the mechanism to fail in constraining overconfident positive updates. As a result, the policy model inevitably enters an implicit over-optimization stage-while the proxy reward continues to increase, essential metrics such as image quality and text-prompt alignment deteriorate sharply, ultimately making the learned policy impractical for real-world use. To address this issue, we introduce GRPO-Guard, a simple yet effective enhancement to existing GRPO frameworks. Our method incorporates ratio normalization, which restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates across denoising timesteps. In addition, a gradient reweighting strategy equalizes policy gradients over noise conditions, preventing excessive updates from particular timestep regions. Together, these designs act as a regulated clipping mechanism, stabilizing optimization and substantially mitigating implicit over-optimization without relying on heavy KL regularization. Extensive experiments on multiple diffusion backbones (e.g., SD3.5M, Flux.1-dev) and diverse proxy tasks demonstrate that GRPO-Guard significantly reduces over-optimization while maintaining or even improving generation quality.

Figures (13)

• Figure 1: Comparison between FlowGRPO and GRPO-Guard under over-optimization. Left: The proxy score and gold score trends during training. As the proxy score increases, FlowGRPO rapidly enters an over-optimization phase, where the gold score continuously declines. Right: A visual comparison between FlowGRPO and GRPO-Guard. Due to severe reward hacking, FlowGRPO suffers from a drastic degradation in diversity, detail richness, visual quality, and text-image consistency (bottom part). In contrast, GRPO-Guard maintains a stable gold score and high visual quality under a comparable proxy score, as shown in the upper part of the figure.
• Figure 2: Comparison of $r(\theta)$ distributions between FlowGRPO and GRPO-Guard across timesteps. (a) Ideally, the ratio distribution should have a mean near 1 and stable variance across timesteps to ensure effective clipping. (b) Under FlowGRPO, the distribution exhibits a leftward mean shift and increasing variance at low-noise timesteps, causing the clipping mechanism to fail—particularly for trajectories with positive advantages. In contrast, GRPO-Guard with RatioNorm preserves a balanced mean and consistent variance (c), enabling proper clipping and stable policy updates across all timesteps.
• Figure 2: Ablation study on major components.
• Figure 3: Gradient magnitude differences across timesteps. In FlowGRPO, gradient magnitudes vary by roughly 20× across timesteps, reflecting the large differences in gradient scale. GRPO-Guard substantially reduces this imbalance, limiting the variation to about 2.5× and preventing over-optimization under any single noise condition.
• Figure 4: Validation curves of proxy scores and gold scores across different training tasks and baseline methods.