Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcing Diffusion Models by Direct Group Preference Optimization

Yihong Luo, Tianyang Hu, Jing Tang

TL;DR

This work tackles the inefficiency of applying policy-gradient reinforcement learning to diffusion models for human-preference alignment. It introduces Direct Group Preference Optimization (DGPO), which directly learns from group-level preferences without a stochastic policy, enabling deterministic ODE rollouts and faster training. By employing advantage-based weighting and a tractable group-reward objective, DGPO leverages fine-grained within-group signals while avoiding the intractable partition function that hinders DPO-based methods. Empirically, DGPO achieves roughly 20× training speedups over Flow-GRPO and state-of-the-art performance on GenEval and robust out-of-domain metrics across compositional image generation, visual text rendering, and human-preference alignment tasks.

Abstract

While reinforcement learning methods such as Group Relative Preference Optimization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost-effective diffusion samplers are based on deterministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, but this reliance on model-agnostic Gaussian noise leads to slow convergence. To resolve this conflict, we propose Direct Group Preference Optimization (DGPO), a new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information of samples within groups. This design eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics. Code is available at https://github.com/Luo-Yihong/DGPO.

Reinforcing Diffusion Models by Direct Group Preference Optimization

TL;DR

This work tackles the inefficiency of applying policy-gradient reinforcement learning to diffusion models for human-preference alignment. It introduces Direct Group Preference Optimization (DGPO), which directly learns from group-level preferences without a stochastic policy, enabling deterministic ODE rollouts and faster training. By employing advantage-based weighting and a tractable group-reward objective, DGPO leverages fine-grained within-group signals while avoiding the intractable partition function that hinders DPO-based methods. Empirically, DGPO achieves roughly 20× training speedups over Flow-GRPO and state-of-the-art performance on GenEval and robust out-of-domain metrics across compositional image generation, visual text rendering, and human-preference alignment tasks.

Abstract

While reinforcement learning methods such as Group Relative Preference Optimization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost-effective diffusion samplers are based on deterministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, but this reliance on model-agnostic Gaussian noise leads to slow convergence. To resolve this conflict, we propose Direct Group Preference Optimization (DGPO), a new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information of samples within groups. This design eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics. Code is available at https://github.com/Luo-Yihong/DGPO.

Paper Structure

This paper contains 35 sections, 18 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Diffusion Models (DMs)
  4. Reward Modeling
  5. RLHF
  6. GRPO Objective shao2024deepseekmath
  7. DPO Objective dpo
  8. Method
  9. Problem Setup
  10. Direct Group Preference Optimization
  11. Advantage-based Weight Design
  12. Timestep Clip Strategy
  13. Experiments
  14. Experimental Setup
  15. Evaluation Tasks
  16. ...and 20 more sections

Figures (7)

  • Figure 1: Our proposed DGPO shows a near 30 times faster training compared to Flow-GRPO on improving GenEval score (Left Figure). The notable improvement is achieved while maintaining strong performance on other out-of-domain metrics (Right Figure).
  • Figure 2: Qualitative comparisons of DGPO against competing methods. It can be seen that our proposed DGPO not only accurately follows the instructions, but also keeps a strong visual quality. All images are generated by the same initial noise.
  • Figure 3: Compare the training speed of Flow-GRPO and our proposed DGPO.
  • Figure 4: Visual comparisons among variants. It can be seen that without the proposed timestep clip strategy, although it can still accurately follow the instruction, the visual quality notably degrades
  • Figure 5: Comparison of visual text rendering across variants.
  • ...and 2 more figures