Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Equilibrium Diffusion Restoration with Parallel Sampling

Jiezhang Cao, Yue Shi, Kai Zhang, Yulun Zhang, Radu Timofte, Luc Van Gool

TL;DR

The paper addresses the inefficiency and opacity of gradient flow in diffusion-model-based image restoration by reframing the sampling process as a deep-equilibrium fixed-point problem, enabling parallel sampling without training. It introduces DeqIR, a DEQ-based framework that solves for the entire sampling chain as a fixed point via Anderson acceleration and allows fast gradients through DEQ inversion. A key contribution is initialization optimization, where gradients through the fixed point guide the initial noise to improve quality and direct generation, all within a zero-shot setting. Experiments across SR, deblurring, inpainting, colorization, and real-world degradations demonstrate superior restoration quality with fewer NFEs, reduced computation, and practical applicability on multi-GPU setups.

Abstract

Diffusion model-based image restoration (IR) aims to use diffusion models to recover high-quality (HQ) images from degraded images, achieving promising performance. Due to the inherent property of diffusion models, most existing methods need long serial sampling chains to restore HQ images step-by-step, resulting in expensive sampling time and high computation costs. Moreover, such long sampling chains hinder understanding the relationship between inputs and restoration results since it is hard to compute the gradients in the whole chains. In this work, we aim to rethink the diffusion model-based IR models through a different perspective, i.e., a deep equilibrium (DEQ) fixed point system, called DeqIR. Specifically, we derive an analytical solution by modeling the entire sampling chain in these IR models as a joint multivariate fixed point system. Based on the analytical solution, we can conduct parallel sampling and restore HQ images without training. Furthermore, we compute fast gradients via DEQ inversion and found that initialization optimization can boost image quality and control the generation direction. Extensive experiments on benchmarks demonstrate the effectiveness of our method on typical IR tasks and real-world settings.

Deep Equilibrium Diffusion Restoration with Parallel Sampling

TL;DR

The paper addresses the inefficiency and opacity of gradient flow in diffusion-model-based image restoration by reframing the sampling process as a deep-equilibrium fixed-point problem, enabling parallel sampling without training. It introduces DeqIR, a DEQ-based framework that solves for the entire sampling chain as a fixed point via Anderson acceleration and allows fast gradients through DEQ inversion. A key contribution is initialization optimization, where gradients through the fixed point guide the initial noise to improve quality and direct generation, all within a zero-shot setting. Experiments across SR, deblurring, inpainting, colorization, and real-world degradations demonstrate superior restoration quality with fewer NFEs, reduced computation, and practical applicability on multi-GPU setups.

Abstract

Diffusion model-based image restoration (IR) aims to use diffusion models to recover high-quality (HQ) images from degraded images, achieving promising performance. Due to the inherent property of diffusion models, most existing methods need long serial sampling chains to restore HQ images step-by-step, resulting in expensive sampling time and high computation costs. Moreover, such long sampling chains hinder understanding the relationship between inputs and restoration results since it is hard to compute the gradients in the whole chains. In this work, we aim to rethink the diffusion model-based IR models through a different perspective, i.e., a deep equilibrium (DEQ) fixed point system, called DeqIR. Specifically, we derive an analytical solution by modeling the entire sampling chain in these IR models as a joint multivariate fixed point system. Based on the analytical solution, we can conduct parallel sampling and restore HQ images without training. Furthermore, we compute fast gradients via DEQ inversion and found that initialization optimization can boost image quality and control the generation direction. Extensive experiments on benchmarks demonstrate the effectiveness of our method on typical IR tasks and real-world settings.
Paper Structure (19 sections, 1 theorem, 13 equations, 10 figures, 5 tables, 2 algorithms)

This paper contains 19 sections, 1 theorem, 13 equations, 10 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep implicit learning (DIL).
  4. Preliminaries
  5. Image restoration.
  6. Methodology
  7. Deep Equilibrium Diffusion Restoration
  8. Initialization Optimization via DEQ Inversion
  9. Experiments
  10. Evaluation on Image Super-Resolution
  11. Evaluation on Image Deblurring
  12. Evaluation on Image Inpainting
  13. Evaluation on Image Colorization
  14. Evaluation on DEQ Inversion
  15. Ablation Study
  16. ...and 4 more sections

Key Result

Proposition 1

(Parallel sampling) Given a degradation matrix ${\bm{A}}$, a degraded image ${\bm y}$ and a Gaussian noise image ${\bm x}_T\sim {\mathcal{N}}({\bf 0}, {\bm{I}})$, for $k \in [1, \ldots, T]$, the state ${\bm x}_{T-k}$ can be predicted by previous states $\{ {\bm x}_{T-k+1}, \ldots, {\bm x}_{T} \}$, i where ${\bm z}_{s}= c_s^0 \hbox{\boldmath $\epsilon$}_{\theta} ({\bm x}_{s}, {s}) + \sqrt{\bar{\alp

Figures (10)

  • Figure 1: Comparisons of different zero-shot DMIR methods in various IR applications on different datasets.
  • Figure 2: Comparisons of sequential sampling and our parallel sampling.
  • Figure 3: Qualitative results of zero-shot $4{\times}$ super-resolution methods on ImageNet and CelabA-HQ.
  • Figure 4: Qualitative results of zero-shot image deblurring (Gaussian) methods.
  • Figure 5: Interesting applications of DEQ inversion.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Proposition 1