Uncovering the Over-smoothing Challenge in Image Super-Resolution: Entropy-based Quantification and Contrastive Optimization

TL;DR

This work identifies the Center-oriented Optimization (COO) problem as a data-property-driven source of over-smoothed results in PSNR-oriented image super-resolution. It formalizes COO using an entropy framework on the conditional high-resolution distribution and proposes an explicit remedy, Detail Enhanced Contrastive Loss (DECLoss), which minimizes the distribution’s variance via patch-level contrastive learning. Empirical results show DECLoss improves perceptual quality for PSNR-based models and achieves state-of-the-art LPIPS while maintaining PSNR, and it also enhances GAN-based SR methods like RaGAN. The approach provides theoretical insight through entropy-based theorems and practical gains across standard SR benchmarks and real-world datasets, highlighting entropy management as a key lever for high-quality SR outputs.

Abstract

PSNR-oriented models are a critical class of super-resolution models with applications across various fields. However, these models tend to generate over-smoothed images, a problem that has been analyzed previously from the perspectives of models or loss functions, but without taking into account the impact of data properties. In this paper, we present a novel phenomenon that we term the center-oriented optimization (COO) problem, where a model's output converges towards the center point of similar high-resolution images, rather than towards the ground truth. We demonstrate that the strength of this problem is related to the uncertainty of data, which we quantify using entropy. We prove that as the entropy of high-resolution images increases, their center point will move further away from the clean image distribution, and the model will generate over-smoothed images. Implicitly optimizing the COO problem, perceptual-driven approaches such as perceptual loss, model structure optimization, or GAN-based methods can be viewed. We propose an explicit solution to the COO problem, called Detail Enhanced Contrastive Loss (DECLoss). DECLoss utilizes the clustering property of contrastive learning to directly reduce the variance of the potential high-resolution distribution and thereby decrease the entropy. We evaluate DECLoss on multiple super-resolution benchmarks and demonstrate that it improves the perceptual quality of PSNR-oriented models. Moreover, when applied to GAN-based methods, such as RaGAN, DECLoss helps to achieve state-of-the-art performance, such as 0.093 LPIPS with 24.51 PSNR on 4x downsampled Urban100, validating the effectiveness and generalization of our approach.

Figures (13)

• Figure 1: The schematic diagram of the COO problem. (a) shows the ideal mapping from low-resolution (LR) to high-resolution (HR). As the number of similar HRs increases, (b) illustrates the actual mapping under the PSNR-oriented methods. The model tends to produce over-smoothed HR images that converge to the center point of similar HRs rather than the ground truth. To address this issue, we introduce the Detail Enhanced Contrastive Loss (DECLoss), depicted in (c), which employs the cluster property of contrastive learning to reduce the influence of the COO problem.
• Figure 2: Illustration of the Center-Oriented Optimization (COO) problem from the geometry perspective. The green arc represents the distribution of potential high-resolution (HR) images $I_y$ corresponding to a given low-resolution (LR) image $I_x$, $D^\dagger I_x$ is the projection point of LR $I_x$ onto HR hyperplane, $\hat{I_y}$ denotes the mean of the HR distribution. Panel (a) shows an extreme case where the entropy of the HR distribution reaches the upper bound of the space. Panel (b) is a general case, each LR pixel has a finite number of potential HR pixels, which is further detailed in panel (c). Panel (d) describes the principle of our proposed method DECLoss, which reduces the variance of the HR distribution using contrastive learning to alleviate the COO problem.
• Figure 3: An illustration of Theorem \ref{['the:entropy-midpoint']}, which states that the entropy $H(P)$ is positive correlated with the number of samples $N$ when $N<\Theta$. The figure also shows that the distance ($L$) between the model output and the blurriest point, decreases monotonically as $N$ increases, and reaches a lower bound when $N \geq \Theta$.
• Figure 4: A toy example of how model size influences the discriminative power. Increasing the number of parameters of the Multi-layer Perceptron model (MLP) improves its discriminative power by making the decision boundary more flexible and adaptive to the data distribution.
• Figure 5: A schematic overview of the DECLoss method. The generated images $\hat{I_y}$ and HR images $I_y$ are first divided into a sequence of flattened patches $\hat{P_y}$ and $P_y$, respectively. Then, patches are classified into positive and negative samples based on their similarities. Finally, contrastive loss $L_d$ is performed on the patches to polarize the details thus reducing the impact of the COO problem ($*$ is an operator represents Eqs. \ref{['eq:cl_sr2hr']} to \ref{['Eq_Q_neg']}).

Theorems & Definitions (4)

• Theorem 1
• proof
• Theorem 2
• proof