Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysis

TL;DR

The paper tackles the computational burden of diffusion-based image generation by revealing that key content changes occur predominantly in the early and late denoising steps through spectral analysis. It introduces Beta Sampling, a distribution-like time-step selection method realized via the Probability Integral Transform, to concentrate steps where low-frequency and high-frequency components shift most. Empirical results on ADM-G and Stable Diffusion show Beta Sampling outperforms uniform sampling in FID and IS and competes with AutoDiffusion in efficiency, with ablations confirming the balanced α=β configuration. The work provides a practical, spectral-guided framework for accelerating diffusion models and suggests avenues for adaptive extensions across architectures and datasets.

Abstract

Generative diffusion models have emerged as a powerful tool for high-quality image synthesis, yet their iterative nature demands significant computational resources. This paper proposes an efficient time step sampling method based on an image spectral analysis of the diffusion process, aimed at optimizing the denoising process. Instead of the traditional uniform distribution-based time step sampling, we introduce a Beta distribution-like sampling technique that prioritizes critical steps in the early and late stages of the process. Our hypothesis is that certain steps exhibit significant changes in image content, while others contribute minimally. We validated our approach using Fourier transforms to measure frequency response changes at each step, revealing substantial low-frequency changes early on and high-frequency adjustments later. Experiments with ADM and Stable Diffusion demonstrated that our Beta Sampling method consistently outperforms uniform sampling, achieving better FID and IS scores, and offers competitive efficiency relative to state-of-the-art methods like AutoDiffusion. This work provides a practical framework for enhancing diffusion model efficiency by focusing computational resources on the most impactful steps, with potential for further optimization and broader application.

Figures (8)

• Figure 1: An Overview. (a) We analyzed the Fourier transform of images generated at each time step during the denoising process and found that the changes in low- and high-frequency components are concentrated in the early and the later stages, respectively. (b) Based on this, we propose a Beta distribution-like sampling method that focuses on key stages with significant frequency changes. (c) Experiments show our method generates higher quality images at lower steps compared to uniform sampling.
• Figure 2: Spectral analysis of denoising process in ADM-G dhariwal2021diffusion_adm and Stable Diffusion (SD) rombach2022ldm. The trend in the changes of high-frequency and low-frequency components during the denoising steps demonstrates that the core of the diffusion model’s image denoising process lies in the early and late stages.
• Figure 3: Probability Density Function (PDF) and Cumulative Distribution Function (CDF) of uniform and Beta distributions.
• Figure 4: Examples generated by ADM-G dhariwal2021diffusion_adm on ImageNet 64$\times$64 with various sampling strategies.
• Figure 5: Examples generated by Stable Diffusion rombach2022ldm with various sampling strategies. The text prompts used for generation are "A man who is wearing a suit and tie" and "Two large elephants are standing beside each other".