Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image Super-resolution Via Latent Diffusion: A Sampling-space Mixture Of Experts And Frequency-augmented Decoder Approach

Feng Luo, Jinxi Xiang, Jun Zhang, Xiao Han, Wei Yang

TL;DR

This paper tackles image super-resolution with latent diffusion models by addressing two key bottlenecks: reconstruction distortion from latent compression and high computational costs. It introduces a Sampling-Space Mixture of Experts to scale the denoising network capacity without prohibitive overhead, and a frequency-compensated decoder to recover high-frequency details lost in latent space. The approach achieves state-of-the-art perceptual quality on 4x blind SR and 8x non-blind SR benchmarks, outperforming several diffusion-based and GAN-based methods while remaining efficient. These innovations advance high-fidelity SR in real-world degradations and large upscaling factors, with practical implications for scalable diffusion-based SR systems.

Abstract

The recent use of diffusion prior, enhanced by pre-trained text-image models, has markedly elevated the performance of image super-resolution (SR). To alleviate the huge computational cost required by pixel-based diffusion SR, latent-based methods utilize a feature encoder to transform the image and then implement the SR image generation in a compact latent space. Nevertheless, there are two major issues that limit the performance of latent-based diffusion. First, the compression of latent space usually causes reconstruction distortion. Second, huge computational cost constrains the parameter scale of the diffusion model. To counteract these issues, we first propose a frequency compensation module that enhances the frequency components from latent space to pixel space. The reconstruction distortion (especially for high-frequency information) can be significantly decreased. Then, we propose to use Sample-Space Mixture of Experts (SS-MoE) to achieve more powerful latent-based SR, which steadily improves the capacity of the model without a significant increase in inference costs. These carefully crafted designs contribute to performance improvements in largely explored 4x blind super-resolution benchmarks and extend to large magnification factors, i.e., 8x image SR benchmarks. The code is available at https://github.com/amandaluof/moe_sr.

Image Super-resolution Via Latent Diffusion: A Sampling-space Mixture Of Experts And Frequency-augmented Decoder Approach

TL;DR

This paper tackles image super-resolution with latent diffusion models by addressing two key bottlenecks: reconstruction distortion from latent compression and high computational costs. It introduces a Sampling-Space Mixture of Experts to scale the denoising network capacity without prohibitive overhead, and a frequency-compensated decoder to recover high-frequency details lost in latent space. The approach achieves state-of-the-art perceptual quality on 4x blind SR and 8x non-blind SR benchmarks, outperforming several diffusion-based and GAN-based methods while remaining efficient. These innovations advance high-fidelity SR in real-world degradations and large upscaling factors, with practical implications for scalable diffusion-based SR systems.

Abstract

The recent use of diffusion prior, enhanced by pre-trained text-image models, has markedly elevated the performance of image super-resolution (SR). To alleviate the huge computational cost required by pixel-based diffusion SR, latent-based methods utilize a feature encoder to transform the image and then implement the SR image generation in a compact latent space. Nevertheless, there are two major issues that limit the performance of latent-based diffusion. First, the compression of latent space usually causes reconstruction distortion. Second, huge computational cost constrains the parameter scale of the diffusion model. To counteract these issues, we first propose a frequency compensation module that enhances the frequency components from latent space to pixel space. The reconstruction distortion (especially for high-frequency information) can be significantly decreased. Then, we propose to use Sample-Space Mixture of Experts (SS-MoE) to achieve more powerful latent-based SR, which steadily improves the capacity of the model without a significant increase in inference costs. These carefully crafted designs contribute to performance improvements in largely explored 4x blind super-resolution benchmarks and extend to large magnification factors, i.e., 8x image SR benchmarks. The code is available at https://github.com/amandaluof/moe_sr.
Paper Structure (14 sections, 6 equations, 7 figures, 8 tables)

This paper contains 14 sections, 6 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Denoising UNet with SS-MOEs
  6. Frequency Compensated Decoder
  7. Experiments
  8. Benchmark Results of 4$\times$ Blind Image SR.
  9. Benchmark Results of 8$\times$ Non-Blind Image SR.
  10. Ablation Studies and Computation Cost Analysis
  11. Conclusion
  12. Appendix
  13. Implement Details
  14. More Qualitative Comparisons

Figures (7)

  • Figure 1: Latent diffusion model for image SR with SS-MoE and frequency augmented decoder.
  • Figure 2: Sampling-Space MoE of the denoising UNet for image SR. "FFN Avg." means averaging the weights of all experts into one FFN for inference after training.
  • Figure 3: The proposed frequency-augmented decoder is conditioned on the low-resolution image and uses the AFF block Huang2023AdaptiveFF to reduce information loss in the frequency domain.
  • Figure 4: Qualitative comparisons on 4$\times$ SR (128 → 512). Our method is capable of achieving better detail consistency and generating realistic texture details. (Zoom in for the best view)
  • Figure 5: Qualitative comparisons on 8$\times$ SR (64 → 512). Our method is capable of generating sharp images with high-fidelity texture details. (Zoom in for the best view)
  • ...and 2 more figures