Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models

TL;DR

Diffusion-NPO tackles the gap in human-preference alignment for diffusion models by targeting not only generating preferred outputs but also avoiding undesired ones via negative preferences. It introduces Negative Preference Optimization (NPO), a plug‑and‑play approach that reuses existing preference-optimization methods to train a negative-preference aligned model, and integrates this with classifier-free guidance through dual weight offsets at inference. The method yields consistent improvements across text-to-image and text-to-video tasks (SD1.5, SDXL, DreamShaper, VideoCrafter2) on metrics such as PickScore, HPSv2, ImageReward, and Laion-Aesthetic, and outperforms training-free CFG-strengthening baselines. By enabling separate optimization of conditional and unconditional outputs, Diffusion-NPO enhances alignment with human preferences while preserving model compatibility and practicality, though it incurs additional storage for dual offsets.

Abstract

Diffusion models have made substantial advances in image generation, yet models trained on large, unfiltered datasets often yield outputs misaligned with human preferences. Numerous methods have been proposed to fine-tune pre-trained diffusion models, achieving notable improvements in aligning generated outputs with human preferences. However, we argue that existing preference alignment methods neglect the critical role of handling unconditional/negative-conditional outputs, leading to a diminished capacity to avoid generating undesirable outcomes. This oversight limits the efficacy of classifier-free guidance~(CFG), which relies on the contrast between conditional generation and unconditional/negative-conditional generation to optimize output quality. In response, we propose a straightforward but versatile effective approach that involves training a model specifically attuned to negative preferences. This method does not require new training strategies or datasets but rather involves minor modifications to existing techniques. Our approach integrates seamlessly with models such as SD1.5, SDXL, video diffusion models and models that have undergone preference optimization, consistently enhancing their alignment with human preferences.

Figures (15)

• Figure 1: Diffusion-NPO enhances high-frequency details, color and lighting, and low-frequency structures in images by aligning human's negative preference.
• Figure 2: High-level overview of negative preference optimization (NPO). (Training) NPO needs no new training strategies and datasets. NPO training can be achieved through switching preference image pairs with existing preference optimization methods. (Inference) NPO trained models serve as the unconditional/negative-conditional predictors in the classifier-free guidance.
• Figure 3: NPO works as a plug-and-play inference enhancement strategy. It can be easily combined with base diffusion models and preference optimized diffusion models for better human preference aligned generation. Zoom out for better comparison in details.
• Figure 4: Plug-and-play NPO on DreamShaper. NPO not only works on the base Stable Diffusion and its preference optimized variants, but also works on improving customized model finetuned on high-quality data.
• Figure 5: Motivating example results of Section \ref{['sec:understanding']}. Applying merged model as the orange component (i.e., prediction for unconditional/negative-conditional inputs) effectively improves the human preference alignment.