DP$^2$O-SR: Direct Perceptual Preference Optimization for Real-World Image Super-Resolution

TL;DR

DP2O-SR tackles real-world image super-resolution by turning diffusion/flow model stochasticity into a supervisory signal. It constructs a hybrid FR/NR perceptual reward and uses multi-sample ranking within each input group, rather than cross-model best-vs-worst comparisons, to align outputs with perceptual preferences without human labels. Hierarchical preference optimization emphasizes informative intra-group and inter-group comparisons, enabling efficient and stable learning across backbones of different capacities. Extensive Real-ISR experiments show significant perceptual improvements and strong generalization to out-of-domain metrics on RealSR, validating the approach for real-world deployment. The framework advances perceptual-controlled SR by leveraging existing IQA models trained on human data and demonstrates robust performance across diffusion and flow backbones with limited supervision.

Abstract

Benefiting from pre-trained text-to-image (T2I) diffusion models, real-world image super-resolution (Real-ISR) methods can synthesize rich and realistic details. However, due to the inherent stochasticity of T2I models, different noise inputs often lead to outputs with varying perceptual quality. Although this randomness is sometimes seen as a limitation, it also introduces a wider perceptual quality range, which can be exploited to improve Real-ISR performance. To this end, we introduce Direct Perceptual Preference Optimization for Real-ISR (DP$^2$O-SR), a framework that aligns generative models with perceptual preferences without requiring costly human annotations. We construct a hybrid reward signal by combining full-reference and no-reference image quality assessment (IQA) models trained on large-scale human preference datasets. This reward encourages both structural fidelity and natural appearance. To better utilize perceptual diversity, we move beyond the standard best-vs-worst selection and construct multiple preference pairs from outputs of the same model. Our analysis reveals that the optimal selection ratio depends on model capacity: smaller models benefit from broader coverage, while larger models respond better to stronger contrast in supervision. Furthermore, we propose hierarchical preference optimization, which adaptively weights training pairs based on intra-group reward gaps and inter-group diversity, enabling more efficient and stable learning. Extensive experiments across both diffusion- and flow-based T2I backbones demonstrate that DP$^2$O-SR significantly improves perceptual quality and generalizes well to real-world benchmarks.

Figures (10)

• Figure 1: (a) Visual results of models trained with FR, NR, and hybrid rewards. FR reward suppresses detail, NR reward encourages hallucinations, while the hybrid reward preserves structure and improves realism. (b) DP²O-SR significantly boosts perceptual quality on the out-of-domain RealSR benchmark realsr, improving both small and large generative Real-ISR models after only 500 steps. Larger models like C-FLUX benefit more from preference supervision.
• Figure 2: Illustration of our DP2O-SR framework. Given the same LR input and noise, a frozen pretrained SR model $\pi_{\text{ref}}$ (blue) generates diverse outputs by varying the random seed. These outputs are first evaluated using our proposed perceptual reward (Sec. 4.1), and then filtered through a perceptual preference curation process (Sec. 4.2) to construct pairwise comparisons. A trainable policy model $\pi_\theta$ (orange), initialized from $\pi_{\text{ref}}$, is subsequently optimized via hierarchical preference optimization (Sec. 4.3), which emphasizes informative comparisons and enhances perceptual alignment.
• Figure 3: Qualitative comparisons of different Real-ISR methods. Please zoom in for a better view.
• Figure 4: Training curves of DP2O-SR on SD2 and FLUX under varying $M$ and $N/M$ configurations. Larger $M$ generally improves reward, while optimal $N/M$ differs across model capacities.
• Figure 5: Perceptual reward statistics by increasing sample count $M$ on (a) C-SD2 and (b) C-FLUX backbones. In both cases, the baseline models (solid) show a widening quality range with more samples, while DP2O-SR (dashed) consistently improves all statistics—especially Worst@M—highlighting stronger robustness and reduced output variability.