Divergence Minimization Preference Optimization for Diffusion Model Alignment

TL;DR

This work addresses the challenge of aligning diffusion models with human preferences without relying on reward modeling by reframing alignment as minimizing a divergence between the learned policy and an optimal, preference-derived policy. It introduces DMPO, which uses reverse KL divergence to push the model toward the dominant modes of human preference, overcoming mean-seeking tendencies of prior approaches like Diffusion-DPO. Theoretical analysis shows DMPO's optimization direction aligns with the RLHF objective under mild conditions, and extensive experiments demonstrate superior performance across base models and tasks, including image generation and editing. The approach provides a principled, scalable pathway to robust preference alignment with practical impact on controllable diffusion-based generation systems.

Abstract

Diffusion models have achieved remarkable success in generating realistic and versatile images from text prompts. Inspired by the recent advancements of language models, there is an increasing interest in further improving the models by aligning with human preferences. However, we investigate alignment from a divergence minimization perspective and reveal that existing preference optimization methods are typically trapped in suboptimal mean-seeking optimization. In this paper, we introduce Divergence Minimization Preference Optimization (DMPO), a novel and principled method for aligning diffusion models by minimizing reverse KL divergence, which asymptotically enjoys the same optimization direction as original RL. We provide rigorous analysis to justify the effectiveness of DMPO and conduct comprehensive experiments to validate its empirical strength across both human evaluations and automatic metrics. Our extensive results show that diffusion models fine-tuned with DMPO can consistently outperform or match existing techniques, specifically consistently outperforming all baseline models across different base models and test sets, achieving the best PickScore in every case, demonstrating the method's superiority in aligning generative behavior with desired outputs. Overall, DMPO unlocks a robust and elegant pathway for preference alignment, bridging principled theory with practical performance in diffusion models.

Figures (10)

• Figure 1: Visualization of the effect of forward KL (DiffusionDPO) and reverse KL (DMPO) alignment on policy learning. We sample from MLP diffusion models trained with DPO and DMPO objectives. Green contours indicate the desirable distribution, while red contours denote the undesirable distribution. Compared to DiffusionDPO (orange samples), DMPO (blue samples) aligns more accurately with the target by concentrating on the dominant mode of the mixture Gaussian, highlighting its stronger alignment capability.
• Figure 2: Qualitative result of different alignment methods. We show the images generated by different models for various prompts which are selected from Pick-a-Pic V2, Parti-Prompt and HPS V2. The top two rows present results based on SD1.5, while the bottom two rows are based on SDXL. "Diff" represents "Diffusion" for simplicity.
• Figure 3: User Study Results. DMPO significantly outperforms all baselines in human evaluation across three evaluation questions.
• Figure 4: Images edited by different models for various prompts which are selected from TEd-bench. DMPO significantly outperforms other baselines in both text alignment and visual quality.
• Figure 5: Images generated by different models (based on SD1.5) for various prompts which are selected from Pick-a-Pic V2, Parti-Prompt and HPS V2.

Theorems & Definitions (2)

• Theorem 1
• Theorem 2