Region-Adaptive Sampling for Diffusion Transformers

TL;DR

This work introduces Region-Adaptive Sampling (RAS), a training-free strategy for Diffusion Transformers that dynamically allocates sampling effort across image regions by updating only the model’s current focus while reusing noise for less-critical areas. Built on the observation that the model’s focus shifts with substantial continuity across timesteps, RAS employs a region score $R_t$ to select fast-update patches and uses caches to maintain global coherence through attention recovery. Through extensive experiments on Stable Diffusion 3 and Lumina-Next-T2I, RAS delivers up to about 2.36x–2.51x speedups with minimal quality loss and achieves comparable human-perceived quality at roughly 1.6x speedup, demonstrating substantial practical gains for real-time diffusion transformers. The approach leverages the DiT’s token flexibility, region-wise masking, and efficient caching to realize significant inference efficiency while preserving prompt fidelity and image quality, marking a meaningful advance in adaptive diffusion model inference.

Abstract

Diffusion models (DMs) have become the leading choice for generative tasks across diverse domains. However, their reliance on multiple sequential forward passes significantly limits real-time performance. Previous acceleration methods have primarily focused on reducing the number of sampling steps or reusing intermediate results, failing to leverage variations across spatial regions within the image due to the constraints of convolutional U-Net structures. By harnessing the flexibility of Diffusion Transformers (DiTs) in handling variable number of tokens, we introduce RAS, a novel, training-free sampling strategy that dynamically assigns different sampling ratios to regions within an image based on the focus of the DiT model. Our key observation is that during each sampling step, the model concentrates on semantically meaningful regions, and these areas of focus exhibit strong continuity across consecutive steps. Leveraging this insight, RAS updates only the regions currently in focus, while other regions are updated using cached noise from the previous step. The model's focus is determined based on the output from the preceding step, capitalizing on the temporal consistency we observed. We evaluate RAS on Stable Diffusion 3 and Lumina-Next-T2I, achieving speedups up to 2.36x and 2.51x, respectively, with minimal degradation in generation quality. Additionally, a user study reveals that RAS delivers comparable qualities under human evaluation while achieving a 1.6x speedup. Our approach makes a significant step towards more efficient diffusion transformers, enhancing their potential for real-time applications.

Figures (11)

• Figure 1: The main subject and the regions with more details are brushed for more steps than other regions in RAS. Each block represents a patchified latent token.
• Figure 2: (a)(b) Accelerating Lumina-Next-T2I and Stable Diffusion 3, with 30 and 28 steps separately. (c)(d) Multiple configurations of RAS outperform rectified flow in both image qualities and text-following. RAS-X stands for RAS with X sampling steps in total. (e) RAS achieves comparable human-evaluation results with the default model configuration while achieving around 1.6x speedup.
• Figure 3: Visualization of predicted noise of each step. DiT model focuses on certain regions during each step and the change in focus is continuous across steps.
• Figure 4: NDCG dcgwang2013theoreticalanalysisndcgtype for each pair of adjacent sampling steps is high throughout the diffusion process, marking the similarities in the ranking of focused tokens ranging from 0 to 1.
• Figure 5: Overview of RAS design. Only current fast-update regions of each step are passed to the model.