Rethinking Direct Preference Optimization in Diffusion Models

TL;DR

This work tackles the mismatch between text-to-image diffusion outputs and human preferences by addressing two key limitations of Direct Preference Optimization (DPO) for diffusion models: insufficient exploration and reward-scale imbalance across diffusion timesteps. It introduces a stable reference-model update with regularization to enable controlled exploration without losing generative prior, and a timestep-aware optimization strategy that emphasizes learning at early, semantically rich timesteps. The approach yields significant improvements over Diffusion-DPO across multiple benchmarks and models (SD1.5, SDXL, SD3), achieving state-of-the-art alignment on several human-preference metrics and maintaining image quality. The method is practical and extensible, with open-source code available, and highlights exploration as a central driver for better alignment in diffusion-based preference optimization.

Abstract

Aligning text-to-image (T2I) diffusion models with human preferences has emerged as a critical research challenge. While recent advances in this area have extended preference optimization techniques from large language models (LLMs) to the diffusion setting, they often struggle with limited exploration. In this work, we propose a novel and orthogonal approach to enhancing diffusion-based preference optimization. First, we introduce a stable reference model update strategy that relaxes the frozen reference model, encouraging exploration while maintaining a stable optimization anchor through reference model regularization. Second, we present a timestep-aware training strategy that mitigates the reward scale imbalance problem across timesteps. Our method can be integrated into various preference optimization algorithms. Experimental results show that our approach improves the performance of state-of-the-art methods on human preference evaluation benchmarks. The code is available at the Github: https://github.com/kaist-cvml/RethinkingDPO_Diffusion_Models.

Figures (13)

• Figure 1: (a) Alignment performance of Diffusion-DPO, baselines, and our proposed method on SD1.5 with PickScore reward. Our method significantly improves the alignment performance over Diffusion-DPO. (b) (solid lines) Implicit reward margin under the reference update strategy, with and without our regularization. (dotted lines) Approximated KL divergence between the training model and the pre-trained model (Diffusion-DPO), and between the reference model and the pre-trained model (ours). (c) Relationship between the divergence from the pre-trained model and the preference score. The illustration shows that controlled divergence enables effective exploration while excessive deviation results in a decline in preference score.
• Figure 2: Imbalance problem in our reference update method. We present the scale of (a) model losses and (b) implicit rewards, (c) the preference accuracy of implicit rewards, (d) and our proposed reward scale schedule $\lambda(t)$.
• Figure 3: Qualitative comparison. We compare the generated outputs from various preference optimization algorithms based on SD1.5, including our method.
• Figure 4: Results of reference model regularization with $\tau \in \{16, 32, 64\}$, evaluated using the PickScore reward.
• Figure 5: Relative increase in divergence with and without reward scale scheduling. In each interval, 100% represents the divergence of our reference update method.

Theorems & Definitions (2)

• Theorem 1
• proof