Towards Self-Improvement of Diffusion Models via Group Preference Optimization
Renjie Chen, Wenfeng Lin, Yichen Zhang, Jiangchuan Wei, Boyuan Liu, Chao Feng, Jiao Ran, Mingyu Guo
TL;DR
This work identifies that pairwise Direct Preference Optimization (DPO) for diffusion models is vulnerable to low-margin preference pairs and data collection costs. It proposes Group Preference Optimization (GPO), which extends DPO to groupwise comparisons and applies reward standardization, enabling self-improvement using online, model-generated data without external annotations. The method introduces a groupwise loss $ ext{GroupLoss} \propto \sum_{i=0}^{G-1} (G-1-2i) \mathbf{s}(x^i,t,\epsilon)$ and standardized rewards $\mathcal{A}_i = \frac{r_i-\overline{r}}{\mathrm{std}(\mathbf{r})}$, achieving stable training and improved performance across counting, text rendering, and text-image alignment on multiple diffusion backbones. Empirical results demonstrate consistent gains over baselines, including strong improvements when combined with YOLO and OCR tasks, and show GPO’s potential as a scalable, data-efficient approach for diffusion-model refinement without external data or heavy annotation requirements.
Abstract
Aligning text-to-image (T2I) diffusion models with Direct Preference Optimization (DPO) has shown notable improvements in generation quality. However, applying DPO to T2I faces two challenges: the sensitivity of DPO to preference pairs and the labor-intensive process of collecting and annotating high-quality data. In this work, we demonstrate that preference pairs with marginal differences can degrade DPO performance. Since DPO relies exclusively on relative ranking while disregarding the absolute difference of pairs, it may misclassify losing samples as wins, or vice versa. We empirically show that extending the DPO from pairwise to groupwise and incorporating reward standardization for reweighting leads to performance gains without explicit data selection. Furthermore, we propose Group Preference Optimization (GPO), an effective self-improvement method that enhances performance by leveraging the model's own capabilities without requiring external data. Extensive experiments demonstrate that GPO is effective across various diffusion models and tasks. Specifically, combining with widely used computer vision models, such as YOLO and OCR, the GPO improves the accurate counting and text rendering capabilities of the Stable Diffusion 3.5 Medium by 20 percentage points. Notably, as a plug-and-play method, no extra overhead is introduced during inference.