Alignment of Diffusion Models: Fundamentals, Challenges, and Future

TL;DR

The paper surveys alignment of diffusion models, detailing how misalignment with human preferences arises and how RLHF, DPO, and test-time alignment address it. It draws inspiration from LLM alignment, surveys data, algorithms, benchmarks, and cross-domain extensions to video, audio, 3D, and scientific applications, and highlights the trade-offs between training-based and test-time approaches. Key contributions include a structured taxonomy of alignment techniques, a compilation of benchmarks and evaluation metrics, and a forward-looking discussion of challenges and future directions such as pluralistic feedback, data-efficient learning, and self-alignment concepts. The work emphasizes the practical impact of reliable, safe, and controllable diffusion models across domains and modalities, guiding researchers and engineers toward robust, human-aligned generative systems.

Abstract

Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions and generate results with undesired properties or even harmful content. Inspired by the success and popularity of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences. This work mainly reviews alignment of diffusion models, covering advancements in fundamentals of alignment, alignment techniques of diffusion models, preference benchmarks, and evaluation for diffusion models. Moreover, we discuss key perspectives on current challenges and promising future directions on solving the remaining challenges in alignment of diffusion models. To the best of our knowledge, our work is the first comprehensive review paper for researchers and engineers to comprehend, practice, and research alignment of diffusion models.

Figures (5)

• Figure 1: Conceptual illustration of diffusion model misalignment and the goal of alignment. A pre-trained model may generate an output that deviates from human intentions or desired qualities (e.g., missing details in a prompt). An aligned model, after an alignment process, produces an output that better reflects human preferences for the same input.
• Figure 2: Statistical overview of research trends. (a) The number of papers on diffusion models at top computer vision conferences (CVPR, ECCV, ICCV) and top machine learning conferences (NeurIPS, ICML, ICLR) since 2017, indicating a growing interest. Note that ECCV and ICCV are held biennially. (b) Comparison of the ratio of papers on LLMs vs. diffusion models (left pie) and the research focus on alignment within these areas (right pie), highlighting the nascent stage of diffusion model alignment.
• Figure 3: The framework of this survey in human alignment of diffusion models and beyond.
• Figure 4: Diffusion models consist of two key processes: a forward diffusion process with a transition kernel $q(x_{t}|x_{t-1})$, where noise is gradually added to a data sample, and a reverse denoising process with a learnable transition kernel $p_{\theta}(x_{t-1}|x_{t})$, where the model learns to denoise Gaussian noise to reconstruct the original data sample.
• Figure 5: The overview of RLHF and DPO of diffusion models.