Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models

TL;DR

This work tackles the mismatch between reinforcement learning from human feedback (RLHF) with policy-gradient GRPO and the deterministic nature of modern flow-matching models. By reinterpreting SDE-based GRPO as distance-driven contrastive learning, it introduces Neighbor GRPO, which constructs a neighborhood of ODE trajectories via initial-noise perturbations and uses a softmax-distance surrogate leaping policy to guide optimization while preserving deterministic ODE inference. The method is bolstered by practical techniques—symmetric anchor sampling, group-wise quasi-norm reweighting, and compatibility with high-order solvers like DPM++—to achieve efficient training and high-quality generation. Empirical results on a FLUX-based flow model show Neighbor GRPO outperforms SDE-based counterparts in training cost, convergence speed, and generation quality, with strong human evaluation backing. The approach offers a scalable, reliable RLHF pathway for high-fidelity, deterministically-sampled visual generation and opens avenues for extending to video synthesis.

Abstract

Group Relative Policy Optimization (GRPO) has shown promise in aligning image and video generative models with human preferences. However, applying it to modern flow matching models is challenging because of its deterministic sampling paradigm. Current methods address this issue by converting Ordinary Differential Equations (ODEs) to Stochastic Differential Equations (SDEs), which introduce stochasticity. However, this SDE-based GRPO suffers from issues of inefficient credit assignment and incompatibility with high-order solvers for fewer-step sampling. In this paper, we first reinterpret existing SDE-based GRPO methods from a distance optimization perspective, revealing their underlying mechanism as a form of contrastive learning. Based on this insight, we propose Neighbor GRPO, a novel alignment algorithm that completely bypasses the need for SDEs. Neighbor GRPO generates a diverse set of candidate trajectories by perturbing the initial noise conditions of the ODE and optimizes the model using a softmax distance-based surrogate leaping policy. We establish a theoretical connection between this distance-based objective and policy gradient optimization, rigorously integrating our approach into the GRPO framework. Our method fully preserves the advantages of deterministic ODE sampling, including efficiency and compatibility with high-order solvers. We further introduce symmetric anchor sampling for computational efficiency and group-wise quasi-norm reweighting to address reward flattening. Extensive experiments demonstrate that Neighbor GRPO significantly outperforms SDE-based counterparts in terms of training cost, convergence speed, and generation quality.

Figures (9)

• Figure 1: GRPO approaches for flow models optimize the sample $x_t$ at each timestep $t$. We revisit it from the perspective of contrastive learning, pushing anchor samples to high-reward candidates and vice versa. Different from SDE-based GRPO approaches xue2025dancegrpoliu2025flowgrpo that conduct sample-wise exploration and optimization, our Neighbor GRPO optimizes the policy at the anchor in a joint force field defined by all candidates in a group. This approach allows full-ODE training, leading to better training efficiency and sample quality.
• Figure 2: Different from SDE-based GRPO approaches, which explore the sample space with noise perturbation defined by SDE, we directly construct a group of similar initial noises, and conduct deterministic ODE sampling.
• Figure 3: Surrogate leaping policy. The ongoing sampling trajectory virtually leaps to another, following the policy distribution: $p_i = \pi_\theta\left(x_t^{(i)} \mid \{s_t\} \right)$.
• Figure 4: Effect of quasi-norm reweighting.
• Figure 5: Training curves towards HPSv2.1.