Accelerating Diffusion-based Super-Resolution with Dynamic Time-Spatial Sampling

TL;DR

This work tackles the high computational cost of diffusion-based image super-resolution by uncovering frequency- and spatial-domain dynamics of HF recovery. It introduces Time-Spatial-aware Sampling (TSS), a training-free framework combining Time Dynamic Sampling (TDS) and Spatial Dynamic Sampling (SDS) to adapt denoising schedules to temporal HF recovery and per-region content. Across six benchmarks, TSS consistently improves perceptual metrics like MUSIQ by 0.2 to 3.0 and achieves state-of-the-art-like results with roughly half the diffusion steps, outperforming existing acceleration methods. The approach is compatible with multiple diffusion SR backbones and offers a practical, broadly applicable pathway for efficient diffusion-based SR, with potential extensions to other diffusion-driven tasks.

Abstract

Diffusion models have gained attention for their success in modeling complex distributions, achieving impressive perceptual quality in SR tasks. However, existing diffusion-based SR methods often suffer from high computational costs, requiring numerous iterative steps for training and inference. Existing acceleration techniques, such as distillation and solver optimization, are generally task-agnostic and do not fully leverage the specific characteristics of low-level tasks like super-resolution (SR). In this study, we analyze the frequency- and spatial-domain properties of diffusion-based SR methods, revealing key insights into the temporal and spatial dependencies of high-frequency signal recovery. Specifically, high-frequency details benefit from concentrated optimization during early and late diffusion iterations, while spatially textured regions demand adaptive denoising strategies. Building on these observations, we propose the Time-Spatial-aware Sampling strategy (TSS) for the acceleration of Diffusion SR without any extra training cost. TSS combines Time Dynamic Sampling (TDS), which allocates more iterations to refining textures, and Spatial Dynamic Sampling (SDS), which dynamically adjusts strategies based on image content. Extensive evaluations across multiple benchmarks demonstrate that TSS achieves state-of-the-art (SOTA) performance with significantly fewer iterations, improving MUSIQ scores by 0.2 - 3.0 and outperforming the current acceleration methods with only half the number of steps.

Figures (11)

• Figure 1: (a) SNR of different frequency components in SUPIR denoising, where high-frequency signals show a unique two-stage pattern. (b) SNR of high-frequency signals in different spatial regions during the denoising process of SUPIR. (c) Noise amplitude in high-frequency regions of SUPIR denoising: higher variance shortens the positive optimization phase. (d) Denoising visualization of a RealPhoto60 sample.
• Figure 2: Overview of the proposed Time-Spatial-aware Sampling.
• Figure 3: Illustration of the Time Dynamic Sampling strategy.
• Figure 4: (a) The Variance Adaptive Smooth Sampling strategy. (b) The common unified embedding injection strategy. (c) The proposed Spatial Dynamic Time Embedding injection strategy.
• Figure 5: Qualitative comparison with state-of-the-art methods. Top: real-world sample from RealPhoto60 datasets. Bottom: synthetic sample from DIV2K valid datasets. More visual results are provided in the full version.