Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervision

TL;DR

DDSPO addresses the data efficiency and granularity limitations of prior diffusion-model preference methods by introducing stepwise, score-space supervision derived from a contrastive policy pair. It replaces explicit human labels or reward models with a practical, prompt-perturbation approach that generates preferred and dispreferred denoising targets from a frozen reference model. The method yields dense per-timestep guidance and demonstrates improvements in text–image alignment and visual quality across multiple backbones (e.g., SD-1.4, SDXL, SANA) while using significantly less supervision. This makes preference-based diffusion fine-tuning more scalable and broadly applicable to practical deployment scenarios. Overall, DDSPO provides a data-efficient, model-agnostic framework for improving alignment and aesthetics in diffusion models without manual annotations or reward modeling.

Abstract

Diffusion models have achieved impressive results in generative tasks such as text-to-image synthesis, yet they often struggle to fully align outputs with nuanced user intent and maintain consistent aesthetic quality. Existing preference-based training methods like Diffusion Direct Preference Optimization help address these issues but rely on costly and potentially noisy human-labeled datasets. In this work, we introduce Direct Diffusion Score Preference Optimization (DDSPO), which directly derives per-timestep supervision from winning and losing policies when such policies are available. Unlike prior methods that operate solely on final samples, DDSPO provides dense, transition-level signals across the denoising trajectory. In practice, we avoid reliance on labeled data by automatically generating preference signals using a pretrained reference model: we contrast its outputs when conditioned on original prompts versus semantically degraded variants. This practical strategy enables effective score-space preference supervision without explicit reward modeling or manual annotations. Empirical results demonstrate that DDSPO improves text-image alignment and visual quality, outperforming or matching existing preference-based methods while requiring significantly less supervision. Our implementation is available at: https://dohyun-as.github.io/DDSPO

Figures (12)

• Figure 1: Toy Experiments Comparison of Diffusion DPO (D-DPO), DSPO and DDSPO. (a) and (b) show samples from the ground-truth distribution and its noisy variant used for training. (c) is generated by the reference diffusion model trained on (b). (d), (e) and (f) are distributions learned by the models finetuned with Diffusion DPO, DSPO and DDSPO, respectively.
• Figure 2: Qualitative Comparison between SDXL and DDSPO. Images are generated from the same prompts and random seeds.
• Figure A: Full Results of Toy Experiments for Comparison of Diffusion DPO (D-DPO), DSPO and DDSPO. Qualitative results under varying dataset size ($N \in \{12, 120, 1200\}$) and regularization strengths ($\beta \in \{200, 400, 800\}$).
• Figure B: Ablation Study on Dataset Size Performance on GenEval (left) and Compbench (right) with various training set sizes. The orange line denotes our method, while the blue line denotes D-DPO.
• Figure C: Ablation Study on $\beta$. Performance on Diffusion DPO (left) and DDSPO (right) with various $\beta$. The blue line denotes Compbench score while the orange line denotes IS.