Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning

TL;DR

This work identifies Preference Mode Collapse (PMC) as a bias-driven loss of diversity when aligning diffusion models with human preferences. It introduces DivGenBench, a diversity-centric benchmark, and proposes Directional Decoupling Alignment (D$^2$-Align), a two-stage optimization framework that first learns a directional correction in reward embeddings and then trains the generator with a corrected reward signal. The approach mitigates reward model biases, enabling both high fidelity and abundant diversity, as demonstrated by comprehensive qualitative, quantitative, and human-evaluation studies. These results suggest a practical pathway to more faithful human-aligned diffusion systems, with implications for robust, diverse generative AI.

Abstract

Recent studies have demonstrated significant progress in aligning text-to-image diffusion models with human preference via Reinforcement Learning from Human Feedback. However, while existing methods achieve high scores on automated reward metrics, they often lead to Preference Mode Collapse (PMC)-a specific form of reward hacking where models converge on narrow, high-scoring outputs (e.g., images with monolithic styles or pervasive overexposure), severely degrading generative diversity. In this work, we introduce and quantify this phenomenon, proposing DivGenBench, a novel benchmark designed to measure the extent of PMC. We posit that this collapse is driven by over-optimization along the reward model's inherent biases. Building on this analysis, we propose Directional Decoupling Alignment (D$^2$-Align), a novel framework that mitigates PMC by directionally correcting the reward signal. Specifically, our method first learns a directional correction within the reward model's embedding space while keeping the model frozen. This correction is then applied to the reward signal during the optimization process, preventing the model from collapsing into specific modes and thereby maintaining diversity. Our comprehensive evaluation, combining qualitative analysis with quantitative metrics for both quality and diversity, reveals that D$^2$-Align achieves superior alignment with human preference.

Figures (15)

• Figure 1: D$^2$-Align breaks the trade-off between human preference and generative diversity, mitigating Preference Mode Collapse (PMC). The top-right plot shows that while baselines struggle with a trade-off—either achieving low diversity or low preference—D$^2$-Align, achieves a state of both higher diversity and higher human preference. The qualitative examples below illustrate this phenomenon. For the same set of varied prompts, baseline methods exhibit severe PMC, generating homogeneous outputs for identity, style, layout, and tone. D$^2$-Align successfully preserves diversity, generating distinct and high-quality images that align with each individual prompt. See Supp. for detail prompts.
• Figure 2: Overview of D$^2$-Align. (Left) Our framework first learns a directional vector ($\boldsymbol{b}_v$) to correct the reward signal while keeping the generator frozen (Stage 1). It then uses this learned direction to guide the optimization of the generator, steering it away from mode collapse (Stage 2). (Right) This visualization shows that while other methods converge to a narrow peak (low diversity), D$^2$-Align finds a superior optimum that balances both quality and generative diversity.
• Figure 3:
• Figure 4: Overview Construction and Evaluation Pipeline of Our DivGenBench. (a) A systematic process for prompt construction. (b) Generation of images across four distinct dimensions based on different prompts. (c) Quantitative evaluation using our four proposed metrics: Identity Divergence Score (IDS), Artistic Style Coverage (ASC), Spatial Dispersion Index (SDI), and Photographic Variance Score (PVS). (d) A comparative analysis positioning our benchmark against existing ones. (e) Detailed benchmark statistics and performance comparison of state-of-the-art methods.
• Figure 5: Training Efficiency and Effectiveness Comparison. D$^2$-Align outperforms baselines by being both more effective and more efficient. It achieves a higher score in fewer steps, whereas methods like DanceGRPO and Flow-GRPO require over 250 steps to attain a similar level of performance.