Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs

Qinpeng Cui, Yixuan Liu, Xinyi Zhang, Qiqi Bao, Qingmin Liao, Li Wang, Tian Lu, Zicheng Liu, Zhongdao Wang, Emad Barsoum

TL;DR

This work introduces DoSSR, a diffusion-based super-resolution framework that leverages pretrained diffusion priors while starting inference from low-resolution inputs. It formalizes a domain shift diffusion process (DoSSR) and its continuous extension (DoS-SDEs), embedding the LR→HR transition into the forward model and deriving efficient reverse-time samplers with domain-shift guidance. By preserving diffusion priors and employing a shifting sequence that enables LR-based initialization, DoSSR achieves strong SR performance with minimal sampling steps (as few as 5), yielding 5–7× faster inference than previous diffusion-prior methods while maintaining or surpassing state-of-the-art perceptual quality. The approach demonstrates favorable results on synthetic and real-world benchmarks, supported by thorough ablations on DoSG, starting time, step count, and solver order. Limitations include seed sensitivity for some images and broader societal considerations around privacy and ethics in high-resolution imaging.

Abstract

Diffusion-based image super-resolution (SR) models have attracted substantial interest due to their powerful image restoration capabilities. However, prevailing diffusion models often struggle to strike an optimal balance between efficiency and performance. Typically, they either neglect to exploit the potential of existing extensive pretrained models, limiting their generative capacity, or they necessitate a dozens of forward passes starting from random noises, compromising inference efficiency. In this paper, we present DoSSR, a Domain Shift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. At the core of our approach is a domain shift equation that integrates seamlessly with existing diffusion models. This integration not only improves the use of diffusion prior but also boosts inference efficiency. Moreover, we advance our method by transitioning the discrete shift process to a continuous formulation, termed as DoS-SDEs. This advancement leads to the fast and customized solvers that further enhance sampling efficiency. Empirical results demonstrate that our proposed method achieves state-of-the-art performance on synthetic and real-world datasets, while notably requiring only 5 sampling steps. Compared to previous diffusion prior based methods, our approach achieves a remarkable speedup of 5-7 times, demonstrating its superior efficiency. Code: https://github.com/QinpengCui/DoSSR.

Taming Diffusion Prior for Image Super-Resolution with Domain Shift SDEs

TL;DR

This work introduces DoSSR, a diffusion-based super-resolution framework that leverages pretrained diffusion priors while starting inference from low-resolution inputs. It formalizes a domain shift diffusion process (DoSSR) and its continuous extension (DoS-SDEs), embedding the LR→HR transition into the forward model and deriving efficient reverse-time samplers with domain-shift guidance. By preserving diffusion priors and employing a shifting sequence that enables LR-based initialization, DoSSR achieves strong SR performance with minimal sampling steps (as few as 5), yielding 5–7× faster inference than previous diffusion-prior methods while maintaining or surpassing state-of-the-art perceptual quality. The approach demonstrates favorable results on synthetic and real-world benchmarks, supported by thorough ablations on DoSG, starting time, step count, and solver order. Limitations include seed sensitivity for some images and broader societal considerations around privacy and ethics in high-resolution imaging.

Abstract

Diffusion-based image super-resolution (SR) models have attracted substantial interest due to their powerful image restoration capabilities. However, prevailing diffusion models often struggle to strike an optimal balance between efficiency and performance. Typically, they either neglect to exploit the potential of existing extensive pretrained models, limiting their generative capacity, or they necessitate a dozens of forward passes starting from random noises, compromising inference efficiency. In this paper, we present DoSSR, a Domain Shift diffusion-based SR model that capitalizes on the generative powers of pretrained diffusion models while significantly enhancing efficiency by initiating the diffusion process with low-resolution (LR) images. At the core of our approach is a domain shift equation that integrates seamlessly with existing diffusion models. This integration not only improves the use of diffusion prior but also boosts inference efficiency. Moreover, we advance our method by transitioning the discrete shift process to a continuous formulation, termed as DoS-SDEs. This advancement leads to the fast and customized solvers that further enhance sampling efficiency. Empirical results demonstrate that our proposed method achieves state-of-the-art performance on synthetic and real-world datasets, while notably requiring only 5 sampling steps. Compared to previous diffusion prior based methods, our approach achieves a remarkable speedup of 5-7 times, demonstrating its superior efficiency. Code: https://github.com/QinpengCui/DoSSR.
Paper Structure (42 sections, 1 theorem, 58 equations, 11 figures, 6 tables)

This paper contains 42 sections, 1 theorem, 58 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Diffusion Process with Domain Shift
  5. Relation to DDPM ho2020denoising.
  6. Relation to ResShift yue2023resshift.
  7. Shifting Sequence.
  8. Diffusion DoS-SDEs
  9. Forward Process.
  10. Reverse Process.
  11. Score Matching.
  12. Solvers for Diffusion DoS-SDEs
  13. Experiments
  14. Experimental setup
  15. Comparisons with State-of-the-Arts
  16. ...and 27 more sections

Key Result

Proposition 3.1

Given an initial value $\bm{x}_s$ at time $s > 0$, the solution $\bm{x}_t$ for the diffusion DoS-SDEs defined in Eq. reverse_SDE at time $t\in[0,s]$ is as follows: where $\lambda_t = \frac{\sigma_t}{\alpha_t(1-\eta_t)}$ and $\bm{z}_s \sim \mathcal{N}(\bm{0}, \bm{I})$.

Figures (11)

  • Figure 1: (a) Latency, MANIQA, and complexity of model comparison on RealLR200wu2023seesr dataset in x4 SR task (for 128×128 LR images). (b) Qualitative comparisons of DoSSR and recent state-of-the-art methods on one typical real-world example. For diffusion-based methods, the suffix "-N" appended to the method name indicates the number of inference steps. Zoom in for a better view.
  • Figure 2: Illustration of the proposed diffusion process with domain shift. (a) In the forward process, we merge the gradual shift from HR to LR domain with standard diffusion process. (b) In the reverse process, we initiate inference from LR domain ($t=t_1$) and use our fast sampler to generate SR images. (c) Comparison of the estimated score between SD and DoSSR. DoSSR inherits the capability of SD in ambient space and enhances learning a pathway from LR to HR domain. (d) The design of the shifting sequence which enables us to initiate inference from $t_1$.
  • Figure 2: Comparison across various selections of starting point $t_1$, evaluated on the DRealSR dataset. The baseline method is DDPM, which employs the original diffusion equation. In all setups, inference is carried out over 5 steps.
  • Figure 3: Qualitative comparisons of different steps of our DoSSR and other diffusion-based SR methods. The "-N" suffix denotes inference steps. Please zoom in for a better view.
  • Figure 3: Comparison of performance of different sampler orders on the DRealSR dataset. In all setups, inference is carried out over 5 steps.
  • ...and 6 more figures

Theorems & Definitions (1)

  • Proposition 3.1: Exact solution of diffusion DoS-SDEs