When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPO

TL;DR

The paper tackles the problem that preference data used to align diffusion-based image generation with human judgments contain minority annotations arising from subjectivity and error. It introduces Adaptive-DPO, a minority-aware framework that uses a self-driven metric combining intra-annotator confidence and inter-annotator stability to identify minority samples and apply instance-wise reweighting and adaptive margins during direct preference optimization. Empirical results across SD1.5 and SDXL backbones on Pick-a-Pic v2 and HPDv2 demonstrate that Adaptive-DPO outperforms prior methods, both on synthetic noisy data and real-world annotations, and generalizes to related approaches like IPO. The work highlights the importance of accounting for subjectivity in preference data to improve the practical alignment of diffusion models with diverse human preferences, enabling more robust and user-aligned image generation.

Abstract

In recent years, the field of image generation has witnessed significant advancements, particularly in fine-tuning methods that align models with universal human preferences. This paper explores the critical role of preference data in the training process of diffusion models, particularly in the context of Diffusion-DPO and its subsequent adaptations. We investigate the complexities surrounding universal human preferences in image generation, highlighting the subjective nature of these preferences and the challenges posed by minority samples in preference datasets. Through pilot experiments, we demonstrate the existence of minority samples and their detrimental effects on model performance. We propose Adaptive-DPO -- a novel approach that incorporates a minority-instance-aware metric into the DPO objective. This metric, which includes intra-annotator confidence and inter-annotator stability, distinguishes between majority and minority samples. We introduce an Adaptive-DPO loss function which improves the DPO loss in two ways: enhancing the model's learning of majority labels while mitigating the negative impact of minority samples. Our experiments demonstrate that this method effectively handles both synthetic minority data and real-world preference data, paving the way for more effective training methodologies in image generation tasks.

Figures (9)

• Figure 1: Two examples from Pick-a-Pic kirstain2024pick. For both rows, the winning images are the right ones.
• Figure 2: Proportion of majority/minority at different flip ratios.
• Figure 3: Here, we add 20% label flip to Pick-a-Pic v2 kirstain2024pick and calculate the metric according to Eq. \ref{['eq:noise_aware_metric_dpo']}. The x axis denotes the interval of the metric and the y axis denote the ratio of noisy samples. We can observes a significant increase of flipped sample ratio as the increase of the metric value.
• Figure 4: Qualitative results with SD1.5 and SDXL as backbone. Please refer to supplementary for the corresponding prompts.
• Figure 5: Qualitative comparison between models finetuned with and without the proposed margin term.