DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers

TL;DR

DDiT addresses the high computational cost of diffusion transformers by dynamically varying patch sizes across denoising steps, guided by the evolving latent manifold. It introduces a test-time patching mechanism with LoRA-enhanced patch embeddings and a lightweight Dynamic Patch Scheduler that uses higher-order latent differences to allocate more computation where needed. The approach yields up to $3.52\times$ speedups on Flux-1.Dev and Wan-2.1 benchmarks with minimal perceptual quality loss, applicable to both text-to-image and text-to-video generation. The results highlight the importance of timestep-aware, content-driven resource allocation for efficient diffusion-based generation.

Abstract

Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to $3.52\times$ and $3.2\times$ speedup on FLUX-1.Dev and Wan $2.1$, respectively, without compromising the generation quality and prompt adherence.

Figures (10)

• Figure 1: DDiT dynamically selects the optimal patch size at each denoising step at inference yielding significant computational gains at no loss of perceptual quality. Results are shown for FLUX-1.Dev flux2023 for text-to-image and Wan-2.1 wan2025 for text-to-video generation. The top panel denotes the baseline (original model), while the remaining panels illustrate outputs from DDiT at different acceleration rates. ImageReward xu2023imagereward, CLIP radford2021learning, and VBench huang2024vbench scores are reported (higher is better).
• Figure 2: Main idea: dynamic tokenization during denoising. Current methods use the same patch size for all denoising steps during inference time. Instead, DDiT adapts the patch size at each timestep according to the latent complexity, allocating fewer tokens for certain timesteps and more tokens for certain others. While DiT divides VAE latents into patches, for illustrative purposes, we use a real image in pixel space.
• Figure 3: Revised patch-embedding layer to support patches of varied resolutions. We modify the standard patch-embedding layer, designed for a fixed patch size $p$, to additionally support patch sizes $p_{\text{new}}$.
• Figure 4: Inference speed vs. patch size. Inference speed measured over $50$ denoising steps for generating $1024\times1024$ images using FLUX-1.Dev flux2023, where every timestep uses a fixed patch size. As the patch size increases from $p$$\rightarrow$$2p$$\rightarrow$$4p$, the number of tokens decreases quadratically (4096 $\rightarrow$ 1024 $\rightarrow$ 256), resulting in approximately $3\times$ and $4\times$ faster inference for $2p$ and $4p$, respectively, compared to $p$.
• Figure 5: Given $\Delta^{(3)}\mathbf{z}_{t-1}$, we divide it into patches of size $p_{i} \times p_{i}$, compute within-patch standard deviation $\boldsymbol\sigma_{t-1}^{p_i}$ of the acceleration.