Scalable Ranked Preference Optimization for Text-to-Image Generation

TL;DR

This work investigates a scalable approach for collecting large-scale and fully synthetic datasets for DPO training and introduces RankDPO to enhance DPO-based methods using the ranking feedback.

Abstract

Direct Preference Optimization (DPO) has emerged as a powerful approach to align text-to-image (T2I) models with human feedback. Unfortunately, successful application of DPO to T2I models requires a huge amount of resources to collect and label large-scale datasets, e.g., millions of generated paired images annotated with human preferences. In addition, these human preference datasets can get outdated quickly as the rapid improvements of T2I models lead to higher quality images. In this work, we investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. Specifically, the preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process, greatly improving the dataset collection efficiency. Moreover, we demonstrate that such datasets allow averaging predictions across multiple models and collecting ranked preferences as opposed to pairwise preferences. Furthermore, we introduce RankDPO to enhance DPO-based methods using the ranking feedback. Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated preference dataset "Syn-Pic" improves both prompt-following (on benchmarks like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user studies). This pipeline presents a practical and scalable solution to develop better preference datasets to enhance the performance of text-to-image models.

Figures (7)

• Figure 1: Our approach, trained on a synthetic preference dataset with a ranking objective in the preference optimization, improves prompt following and visual quality for SDXL podell2023sdxl and SD3-Medium sd3, without requiring any manual annotations.
• Figure 2: Overview of our two novel components: (A) Syn-Pic and (B) RankDPO. Left illustrates the pipeline to generate a synthetically ranked preference dataset. It starts by collecting prompts and generating images using the same prompt for different T2I models. Next, we calculate the overall preference score using Reward models (e.g., PickScore, ImageReward). Finally, we rank these images in the decreasing order of preference scores. Right: Given true preference rankings for generated images per prompt, we first obtain predicted ranking by current model checkpoint using scores ${\mathbf{s}}_i$ (see Eq. \ref{['eq.rankdpo_scores_i']}). In this instance, although the predicted ranking is inverse of the true rankings, the ranks $(1,4)$ obtains a larger penalty than the ranks $(2,3)$. This penalty is added to our ranking loss through DCG weights (see Eq. \ref{['eq.dcg_weights_delta_ij']}). Thus, by optimizing $\bm{\theta}$ with Ranking Loss (see Eq. \ref{['eq.rank_dpo_objective']}), the updated model addresses the incorrect rankings $(1,4)$. This procedure is repeated over the training process, where the rankings induced by the model aligns with the labelled preferences.
• Figure 3: Win rates of our approach compared to DPO-SDXL and SDXL on human evaluation.
• Figure 4: Comparison among different preference optimization methods and RankDPO for SDXL. The results illustrate that we generate images with better prompt alignment and aesthetic quality.
• Figure 5: Comparison among different preference optimization methods and RankDPO for SDXL. The results illustrate that we generate images with better prompt alignment and visual quality.