One-Step Effective Diffusion Network for Real-World Image Super-Resolution

TL;DR

It is argued that the LQ image contains rich information to restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling.

Abstract

The pre-trained text-to-image diffusion models have been increasingly employed to tackle the real-world image super-resolution (Real-ISR) problem due to their powerful generative image priors. Most of the existing methods start from random noise to reconstruct the high-quality (HQ) image under the guidance of the given low-quality (LQ) image. While promising results have been achieved, such Real-ISR methods require multiple diffusion steps to reproduce the HQ image, increasing the computational cost. Meanwhile, the random noise introduces uncertainty in the output, which is unfriendly to image restoration tasks. To address these issues, we propose a one-step effective diffusion network, namely OSEDiff, for the Real-ISR problem. We argue that the LQ image contains rich information to restore its HQ counterpart, and hence the given LQ image can be directly taken as the starting point for diffusion, eliminating the uncertainty introduced by random noise sampling. We finetune the pre-trained diffusion network with trainable layers to adapt it to complex image degradations. To ensure that the one-step diffusion model could yield HQ Real-ISR output, we apply variational score distillation in the latent space to conduct KL-divergence regularization. As a result, our OSEDiff model can efficiently and effectively generate HQ images in just one diffusion step. Our experiments demonstrate that OSEDiff achieves comparable or even better Real-ISR results, in terms of both objective metrics and subjective evaluations, than previous diffusion model-based Real-ISR methods that require dozens or hundreds of steps. The source codes are released at https://github.com/cswry/OSEDiff.

Figures (7)

• Figure 1: Performance and efficiency comparison among SD-based Real-ISR methods. (a). Performance comparison on the DrealSR benchmark drealsr. Metrics like LPIPS and NIQE, where smaller scores indicate better image quality, are inverted and normalized for display. OSEDiff achieves leading scores on most metrics with only one diffusion step. (b). Model efficiency comparison. The inference time is tested on an A100 GPU with $512 \times 512$ input image size. OSEDiff has the fewest trainable parameters and is over 100 times faster than StableSR wang2024exploiting.
• Figure 2: The training framework of OSEDiff. The LQ image is passed through a trainable encoder $\emph{E}_\theta$, a LoRA finetuned diffusion network $\boldsymbol{\epsilon}_\theta$ and a frozen decoder $\emph{D}_\theta$ to obtain the desired HQ image. In addition, text prompts are extracted from the LQ image and input to the diffusion network to stimulate its generation capacity. Meanwhile, the output of the diffusion network $\boldsymbol{\epsilon}_\theta$ will be sent to two regularizer networks (a frozen pre-trained one and a fine-tuned one), where variational score distillation is performed in latent space to ensure that the output of $\boldsymbol{\epsilon}_\theta$ follows HQ natural image distribution. The regularization loss will be back-propagated to update $\emph{E}_\theta$ and $\boldsymbol{\epsilon}_\theta$. Once training is finished, only $\emph{E}_\theta$, $\boldsymbol{\epsilon}_\theta$ and $\emph{D}_\theta$ will be used in inference.
• Figure 3: Qualitative comparisons of different Real-ISR methods. Please zoom in for a better view.
• Figure 4: The impact of different prompt extraction methods. Please zoom in for a better view.
• Figure 5: Qualitative comparisons between OSEDiff and GAN-based Real-ISR methods. Please zoom in for a better view.