Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback

TL;DR

Diffusion-DRO reframes diffusion-model preference learning as a max-margin inverse reinforcement learning problem, removing reliance on a reward model and paired comparisons by using expert demonstrations and online negatives. By introducing trajectory rewards and a thresholded ranking loss, it yields a stable, single-stage objective that directly optimizes the margin between expert and policy trajectories. Empirically, Diffusion-DRO outperforms strong baselines across Pick-a-Pic v2 and HPDv2 benchmarks and gains strong human preference signals in MTurk studies, demonstrating robust generalization to unseen prompts. This approach offers a scalable, data-efficient path to aligning diffusion models with nuanced human aesthetics and preferences, with open-source code and pre-trained models available.

Abstract

Direct preference optimization (DPO) methods have shown strong potential in aligning text-to-image diffusion models with human preferences by training on paired comparisons. These methods improve training stability by avoiding the REINFORCE algorithm but still struggle with challenges such as accurately estimating image probabilities due to the non-linear nature of the sigmoid function and the limited diversity of offline datasets. In this paper, we introduce Diffusion Denoising Ranking Optimization (Diffusion-DRO), a new preference learning framework grounded in inverse reinforcement learning. Diffusion-DRO removes the dependency on a reward model by casting preference learning as a ranking problem, thereby simplifying the training objective into a denoising formulation and overcoming the non-linear estimation issues found in prior methods. Moreover, Diffusion-DRO uniquely integrates offline expert demonstrations with online policy-generated negative samples, enabling it to effectively capture human preferences while addressing the limitations of offline data. Comprehensive experiments show that Diffusion-DRO delivers improved generation quality across a range of challenging and unseen prompts, outperforming state-of-the-art baselines in both both quantitative metrics and user studies. Our source code and pre-trained models are available at https://github.com/basiclab/DiffusionDRO.

Figures (6)

• Figure 1: User study results comparing Diffusion-DRO with baseline methods. The win rate represents the proportion of survey questions where users preferred Diffusion-DRO over the baselines.
• Figure 2: From top to bottom, the text prompts are: "A Pixar lemon wearing sunglasses on a beach," "A dragon sitting on a couch in a digital illustration," "A detailed painting of Atlantis by multiple artists, featuring intricate detailing and vibrant colors," and "A passenger jet aircraft flying in the sky."
• Figure 3: Evaluation results of Diffusion-DRO and SFT trained with varying amounts of expert demonstrations.
• Figure 4: The prompts used for image generation are sourced from the HPDv2 Benchmark, categorized as Anime, Concept Art, Painting, and Photo from top to bottom, respectively. The specific prompts, in order, are: "A portrait of a smiling Dragonite in a sunflower field with a cloudy sky backdrop," "Amphitheater filled with crowd looking at a dumpster on fire in patriotic colors," "Beige canvas tents set up in an arctic landscape with no vegetation, surrounded by rolling hills - reminiscent of a romanticist painting," and "A small bird sitting in a metal wheel."
• Figure 5: The prompts used for image generation are sourced from the HPDv2 Benchmark, categorized as Anime, Concept Art, Painting, and Photo from top to bottom, respectively. The specific prompts, in order, are: "A monkey wearing a jacket," "Portrait of a cyberpunk gang," "Bob Ross painting Mario on an easel in his office," and "A little girl holding a brown stuffed animal."