E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Models

TL;DR

This work addresses sparse and ambiguous reward signals in GRPO-based training for flow models by revealing that high-entropy denoising steps drive meaningful exploration. It introduces E-GRPO, which adaptively merges consecutive low-entropy steps into single high-entropy steps while applying ODE sampling on remaining steps, and uses multi-step group normalized advantages to provide dense, reliable credit assignment. The approach is supported by entropy analysis and empirical results showing state-of-the-art performance on the HPS metric and improvements on out-of-domain rewards under multi-reward settings, validating the effectiveness of entropy-guided stochastic optimization for visual generation. Overall, E-GRPO improves training efficiency, stability, and human-preference alignment for flow-based models, while mitigating reward hacking through robust reward signaling.

Abstract

Recent reinforcement learning has enhanced the flow matching models on human preference alignment. While stochastic sampling enables the exploration of denoising directions, existing methods which optimize over multiple denoising steps suffer from sparse and ambiguous reward signals. We observe that the high entropy steps enable more efficient and effective exploration while the low entropy steps result in undistinguished roll-outs. To this end, we propose E-GRPO, an entropy aware Group Relative Policy Optimization to increase the entropy of SDE sampling steps. Since the integration of stochastic differential equations suffer from ambiguous reward signals due to stochasticity from multiple steps, we specifically merge consecutive low entropy steps to formulate one high entropy step for SDE sampling, while applying ODE sampling on other steps. Building upon this, we introduce multi-step group normalized advantage, which computes group-relative advantages within samples sharing the same consolidated SDE denoising step. Experimental results on different reward settings have demonstrated the effectiveness of our methods.

Figures (8)

• Figure 1: The influence of entropy for sampling results. (a) We visualize the generation images with different SDE sampling strategy, including one-step SDE on step 2, one-step SDE on step 6, and merged-step SDE on step 6. We also report the variance of clip score for generated images. Samples from the initial steps and merged steps share higher differences, while posterior steps generate undistinguishable samples, whose variance is similar to small perturbation on original images. (b) We report the entropy of SDE sampling on each timestep with different merged steps. More merged steps indicate higher entropy and larger exploration scope in RL training. (c) We visualize the training reward curves on models trained on all timesteps, the first half timesteps, and the second half timesteps.
• Figure 2: E-GRPO sampling strategy. First, we generate a set of anchor noise latents corresponding to different timesteps. For each active SDE timestep $t_i$, merged steps $\mathcal{T}_i$ is selected based on entropy analysis. We generate a group of results based on the specific SDE sampling of merged steps, and compute the advantage within each group.
• Figure 3: Ambiguous reward signal. For consecutive multi-step SDE sampling, the advantage is corresponding to multiple timesteps, which may results in wrong optimization direction on the specific timestep. Our merged-step SDE sampling not only enlarges the exploration scope, but also eliminate ambiguous reward by aligning the final advantage to one merged SDE step.
• Figure 4: Comparison of Training Reward Curves. The reward curve of E-GRPO demonstrates faster and more stable improvement during training compared to baseline methods. This indicates that exploration guided by high-entropy steps can enhance learning efficiency while mitigating noise in the reward signal.
• Figure 5: Visualization Comparisons. Comparison between E-GRPO with other baseline methods. E-GRPO better integrates semantics and fine-grained details.