SDiT: Semantic Region-Adaptive for Diffusion Transformers

TL;DR

This work tackles the computational bottlenecks of diffusion transformers by introducing SDiT, a training-free framework that allocates denoising effort according to semantic region complexity. It combines a fast, semantic-based Quickshift clustering, a complexity-driven regional scheduler with boundary-aware refinement, and a velocity-space extrapolation mechanism to handle non-uniform timesteps. The approach achieves up to $3.0\times$ end-to-end speedup with minimal degradation in perceptual and semantic quality, outperforming uniform region strategies like RAS and preserving fine details and coherence at high resolutions (e.g., $1024\times1024$). By operating without retraining or architectural changes, SDiT offers a practical, structure-aware route toward efficient diffusion-based image synthesis with broad deployment potential.

Abstract

Diffusion Transformers (DiTs) achieve state-of-the-art performance in text-to-image synthesis but remain computationally expensive due to the iterative nature of denoising and the quadratic cost of global attention. In this work, we observe that denoising dynamics are spatially non-uniform-background regions converge rapidly while edges and textured areas evolve much more actively. Building on this insight, we propose SDiT, a Semantic Region-Adaptive Diffusion Transformer that allocates computation according to regional complexity. SDiT introduces a training-free framework combining (1) semantic-aware clustering via fast Quickshift-based segmentation, (2) complexity-driven regional scheduling to selectively update informative areas, and (3) boundary-aware refinement to maintain spatial coherence. Without any model retraining or architectural modification, SDiT achieves up to 3.0x acceleration while preserving nearly identical perceptual and semantic quality to full-attention inference.

Figures (12)

• Figure 1: SDiT accelerates image generation while maintaining high fidelity. We introduce SDiT, a training-free algorithm that accelerates diffusion transformer inference by dynamically allocating computation across semantically coherent regions, achieving substantial speedup while preserving image fidelity. Under the same model and step budget, our method achieves a 2.70× speedup with sharp details and good boundary preservation. Even on the most challenging cases—such as single or multi-face generation, SDiT consistently preserves fine details and structural coherence. In contrast, RAS struggles to maintain pixel fidelity (e.g., blurred or collapsed facial textures) across object regions. The rightmost columns visualize SDiT’s segmentation and complexity maps which guide adaptive region-wise denoising. Setting: Lumina-Next with 30-step, 1024 × 1024 resolution. Latency is measured on Ada6000.
• Figure 2: Visualization of the diffusion process
• Figure 3: Overview of Diffusion Transformer Structure.
• Figure 4: Average attention strength as a function of Chebyshev distance across denoising steps.
• Figure 5: Attention heatmaps from the center patch (marked by a blue star) across denoising steps.