Self-NPO: Data-Free Diffusion Model Enhancement via Truncated Diffusion Fine-Tuning

TL;DR

Self-NPO introduces a data-free negative preference optimization framework for diffusion models by learning from the model's own outputs through truncated diffusion fine-tuning. It leverages Tweedie's formula to perform partial denoising, enabling efficient training while preserving the target distribution to reduce undesirable outputs via classifier-free guidance. The approach is theoretically grounded via three theorems connecting distributions, gradients, and standard diffusion training, and empirically validated on SD1.5, SDXL, and CogVideoX with substantial gains in generation quality and human-alignment metrics at a fraction of the training cost of prior methods. This work offers a practical, plug-and-play solution for aligning diffusion models with human preferences in data-scarce domains, with open-source code available.

Abstract

Diffusion models have demonstrated remarkable success in various visual generation tasks, including image, video, and 3D content generation. Preference optimization (PO) is a prominent and growing area of research that aims to align these models with human preferences. While existing PO methods primarily concentrate on producing favorable outputs, they often overlook the significance of classifier-free guidance (CFG) in mitigating undesirable results. Diffusion-NPO addresses this gap by introducing negative preference optimization (NPO), training models to generate outputs opposite to human preferences and thereby steering them away from unfavorable outcomes through CFG. However, prior NPO approaches rely on costly and fragile procedures for obtaining explicit preference annotations (e.g., manual pairwise labeling or reward model training), limiting their practicality in domains where such data are scarce or difficult to acquire. In this work, we propose Self-NPO, specifically truncated diffusion fine-tuning, a data-free approach of negative preference optimization by directly learning from the model itself, eliminating the need for manual data labeling or reward model training. This data-free approach is highly efficient (less than 1% training cost of Diffusion-NPO) and achieves comparable performance to Diffusion-NPO in a data-free manner. We demonstrate that Self-NPO integrates seamlessly into widely used diffusion models, including SD1.5, SDXL, and CogVideoX, as well as models already optimized for human preferences, consistently enhancing both their generation quality and alignment with human preferences. Code is available at https://github.com/G-U-N/Diffusion-NPO.

Figures (8)

• Figure 1: Motivation of Self-NPO. Preference Optimization (PO) optimizes the preference aligned model by training on preference annotations. Negative Preference Optimization (NPO) optimizes the negative preference aligned model by training on switched preference annotations. Self-NPO optimizes the negative preference aligned model by truncated diffusion fine-tuning (TDFT) on truncated self-generated data.
• Figure 2: Visual Comparison. Prompts from left to right, from top to bottom are "galaxy cat", "cat as a mafia boss", and "A real picture of an Canguru with sunglasses at the sunset holding a stop sign".
• Figure 3: Visual comparison on Stable Diffusion XL. Prompts: "A cute blue cat.", "An attractive young woman rolling her eyes.", and "Close-up of a head with smoke from ears, watching a smartphone, dynamic angle".
• Figure 4: Truncated diffusion fine-tuning (TDFT). Baseline method requires full simulation of generation process, introducing huge amount of generation costs. Naive truncation reduces the simulation costs but introduces distribution mismatch. TDFT reduces the simulation costs and maintains the target distribution.
• Figure 5: Quantitative performance comparison with stable diffusion XL based models. All metrics are tested with official weights.