Dual Caption Preference Optimization for Diffusion Models

TL;DR

This work tackles the limited supervision signal in diffusion-model alignment caused by semantic overlap and irrelevant prompts between preferred and less-preferred image prompts. It introduces Dual Caption Preference Optimization (DCPO), a data-augmentation framework with three variants—DCPO-c (captioning), DCPO-p (perturbation), and DCPO-h (hybrid)—that assign two distinct captions $z^w$ and $z^l$ to each preference pair and optimize the diffusion model via a dual-caption objective that increases $p_ heta(x^w|z^w)$ while decreasing $p_ heta(x^l|z^l)$. To enable this, the Pick-Double Caption dataset is constructed by generating two captions per image from captioning models (e.g., LLaVA, Emu2) or perturbations of the original prompt, thereby expanding the supervision signal and reducing semantic overlap. Empirically, DCPO variants outperform Stable Diffusion 2.1 fine-tuned baselines (SFT_Chosen, Diffusion-DPO, MaPO) across multiple metrics (Pickscore, HPSv2.1, GenEval, CLIPscore, ImageReward) and show favorable GPT-4o judgments on PartiPrompts, with DCPO-h delivering the strongest gains. The method demonstrates robust improvements across in-distribution data, with clear evidence that more correlated captions and controlled perturbations yield better alignment, albeit at significant pre-processing cost; future work may focus on efficiency and broader backbone support, including safety-related applications.

Abstract

Recent advancements in human preference optimization, originally developed for Large Language Models (LLMs), have shown significant potential in improving text-to-image diffusion models. These methods aim to learn the distribution of preferred samples while distinguishing them from less preferred ones. However, within the existing preference datasets, the original caption often does not clearly favor the preferred image over the alternative, which weakens the supervision signal available during training. To address this issue, we introduce Dual Caption Preference Optimization (DCPO), a data augmentation and optimization framework that reinforces the learning signal by assigning two distinct captions to each preference pair. This encourages the model to better differentiate between preferred and less-preferred outcomes during training. We also construct Pick-Double Caption, a modified version of Pick-a-Pic v2 with separate captions for each image, and propose three different strategies for generating distinct captions: captioning, perturbation, and hybrid methods. Our experiments show that DCPO significantly improves image quality and relevance to prompts, outperforming Stable Diffusion (SD) 2.1, SFT_Chosen, Diffusion-DPO, and MaPO across multiple metrics, including Pickscore, HPSv2.1, GenEval, CLIPscore, and ImageReward, fine-tuned on SD 2.1 as the backbone.

Figures (16)

• Figure 1: Sample images generated by different methods on the HPSv2, Geneval, and Pickscore benchmarks. After fine-tuning SD 2.1 with $\text{SFT}_{\text{Chosen}}$, Diffusion-DPO, MaPO, and DCPO on Pick-a-Picv2 and Pick-Double Caption datasets, DCPO produces images with notably higher preference and visual appeal (See more examples in Appendix \ref{['sec:app_additional_examples']}).
• Figure 2: The DCPO pipeline in 3 variants: DCPO-c, DCPO-p, and DCPO-h, all of which require a duo of a captioned preferred image $(x^w_0, z^w)$ and a captioned less-preferred image $(x^l_0, z^l)$. DCPO-c (Top Left): We use a captioning model to generate distinctive captions respectively for images $x^w_0$ and $x^l_0$ given the shared prompt $c$. DCPO-p (Bottom Left): We take prompt $c$ as the caption for image $x^w_0$, then we use a Large Language Model (LLM) to generate a semantically perturbed prompt $z_p^l$ given prompt $c$ as the caption for image $x^l_0$. DCPO-h (Right): A hybrid method where the generated caption $z^l$ is now perturbed into $z_p^l$ for image $x^l_0$. Our Pick-Double Caption Dataset discussed in Section \ref{['sec:pick-double-caption']} is constructed using DCPO-c.
• Figure 3: The semantic overlap issue in the Pick-a-Pic v2 dataset. $\mu^l$ and $\mu^w$ represent the average CLIPscore of preferred and less preferred images for prompt $c$, respectively. Also, $\Delta \mu$ shows the difference between the distributions.
• Figure 4: Effect of the perturbation method on semantic distributions in terms of CLIPScore. (a) shows the distributions that feature the captions $z^w$ and $z^l$ generated by the LLaVA model, while (b), (c), and (d) represent different levels of perturbation on top of the caption $z^l$. The figure demonstrates that as the level of perturbation increases, the distance between the distributions of captions $z^w$ and $z^l$ increases. For more details on the perturbation method, refer to Appendix \ref{['sec:appendix_perturbation']}.
• Figure 5: Performance comparison of DCPO-c and DCPO-h on different perturbation levels. We plotted regression lines for the four models, showing that as $\Delta \mu$ increases, performance improves but drops after a threshold $t$ (orange boundary).