Perception-Distortion Balanced Super-Resolution: A Multi-Objective Optimization Perspective

TL;DR

This paper forms the perception-distortion trade-off in SR as a multi-objective optimization problem and develops a new optimizer by integrating the gradient-free evolutionary algorithm (EA) with gradient-based Adam, where EA and Adam focus on the divergence and convergence of the optimization directions respectively.

Abstract

High perceptual quality and low distortion degree are two important goals in image restoration tasks such as super-resolution (SR). Most of the existing SR methods aim to achieve these goals by minimizing the corresponding yet conflicting losses, such as the $\ell_1$ loss and the adversarial loss. Unfortunately, the commonly used gradient-based optimizers, such as Adam, are hard to balance these objectives due to the opposite gradient decent directions of the contradictory losses. In this paper, we formulate the perception-distortion trade-off in SR as a multi-objective optimization problem and develop a new optimizer by integrating the gradient-free evolutionary algorithm (EA) with gradient-based Adam, where EA and Adam focus on the divergence and convergence of the optimization directions respectively. As a result, a population of optimal models with different perception-distortion preferences is obtained. We then design a fusion network to merge these models into a single stronger one for an effective perception-distortion trade-off. Experiments demonstrate that with the same backbone network, the perception-distortion balanced SR model trained by our method can achieve better perceptual quality than its competitors while attaining better reconstruction fidelity. Codes and models can be found at https://github.com/csslc/EA-Adam}{https://github.com/csslc/EA-Adam.

Figures (11)

• Figure 1: Performance comparison of perception-distortion balanced SR models trained by the Adam optimizer and our hybrid EA-Adam optimizer on the BSD100 dataset. For the perceptual quality index LPIPS, the smaller the better. For the distortion degree index PSNR, the higher the better. The models optimized by Adam (orange triangle points) with different weighted combinations of $\ell_1$, perceptual and adversarial losses are either stagnated into an unsatisfactory solution, or achieve unbalanced performance. The models optimized by our EA-Adam optimizer (blue circle points) can obtain better convergence and divergence. Furthermore, the fused model (red star) from EA-Adam models achieves the best perceptual quality index while maintaining a high fidelity index.
• Figure 2: (a) The framework of traditional GAN-based SR methods. (b) The Adam steps optimize a set of generator and discriminator networks. (c) The EA steps optimize the generator networks by fixing the discriminator networks.
• Figure 3: Illustration of the convergence process of the proposed EA-Adam optimizer in an EA-Adam cycle ($T_{Adam}+T_{EA}$). The Adam steps primarily focus on optimization convergence, enabling rapid search for improved objective losses. In contrast, the EA step emphasizes divergence optimization, ensuring a uniform spread of solutions across the plane.
• Figure 4: Two stages of our network fusion process. Left: training of the attention-based weight regression network, which predicts the fusion weights of experts for each input image. Right: model fusion by averaging weight vectors of a batch of validation data.
• Figure 5: Visual comparisons among the proposed EA-Adam optimization method and other state-of-the-art methods under the SRResNet backbone.