ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Models

TL;DR

This work addresses the tension between perceptual quality and diversity when aligning diffusion-based image generators with human preference signals. It introduces combined generation, which preserves early-stage global structure by using the base model in initial diffusion steps and the reward-tuned model only in later steps, and ImageReFL, a training scheme that refines final steps using real images and multiple regularizers to mitigate overfitting and reward hacking. Empirical results across multiple models and reward metrics show improved diversity without sacrificing quality, and a user study corroborates better human alignment and variety. The approach offers a practical path toward more human-aligned, diverse image synthesis, albeit at the cost of maintaining two model weights during inference and depending on reward signal fidelity.

Abstract

Recent advances in diffusion models have led to impressive image generation capabilities, but aligning these models with human preferences remains challenging. Reward-based fine-tuning using models trained on human feedback improves alignment but often harms diversity, producing less varied outputs. In this work, we address this trade-off with two contributions. First, we introduce \textit{combined generation}, a novel sampling strategy that applies a reward-tuned diffusion model only in the later stages of the generation process, while preserving the base model for earlier steps. This approach mitigates early-stage overfitting and helps retain global structure and diversity. Second, we propose \textit{ImageReFL}, a fine-tuning method that improves image diversity with minimal loss in quality by training on real images and incorporating multiple regularizers, including diffusion and ReFL losses. Our approach outperforms conventional reward tuning methods on standard quality and diversity metrics. A user study further confirms that our method better balances human preference alignment and visual diversity. The source code can be found at https://github.com/ControlGenAI/ImageReFL .

Figures (19)

• Figure 1: Diversity degradation during ReFL training on HPSv2.1 using SD1.5. The first row shows outputs from the original model, which produces diverse but less appealing images. As training progresses (second and third rows), visual appeal improves, but diversity collapses, with increasingly similar outputs across prompts.
• Figure 2: Image quality and diversity metrics during ReFL training on SD1.5 with HPSv2.1 as the target reward. HPSv2.1 indicates improvements in visual quality, while DinoDiversity, FID, and LogCovDistance capture the decline in diversity over training iterations.
• Figure 3: Comparison of generations by ReFL, ReFL Combined, and ImageReFL. ReFL achieves high visual quality but suffers from low diversity. ReFL Combined improves diversity by using the base model in early steps, while ImageReFL further enhances the quality–diversity trade-off.
• Figure 4: Demonstration of the combined generation method applied to different base models (Stable Diffusion 1.5 and SDXL) and target reward functions (HPSv2.1 and PickScore).
• Figure 5: Image quality and diversity trade-off for SD1.5 trained on the HPSv2.1 score using combined generation. MPS reflects image quality, while DinoDiversity, FID, and LogCovDistance measure diversity.