Sortblock: Similarity-Aware Feature Reuse for Diffusion Model

TL;DR

Diffusion Transformers suffer from slow inference due to the sequential denoising process. The authors propose SortBlock, a training-free acceleration framework that adaptively caches block-level DiT features by evaluating the similarity of input-output changes across adjacent timesteps and recomputing only the most change-prone blocks; to mitigate error accumulation, a lightweight linear prediction reuses and predicts skipped features. The method uses stage-aware reuse and two hyperparameters, the policy refresh interval $K$ and the adaptive update ratio $\rho$, to balance speed and fidelity. Experiments on Flux.1-dev, Wan2.1, and HunyuanVideo show about 2× speedups with minimal degradation and superior performance to other training-free baselines across image and video tasks. SortBlock offers a general, training-free route to practical diffusion-based generation at higher resolutions and longer sequences.

Abstract

Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities, particularly benefiting from Transformer architectures that enhance visual and artistic fidelity. However, their inherently sequential denoising process results in high inference latency, limiting their deployment in real-time scenarios. Existing training-free acceleration approaches typically reuse intermediate features at fixed timesteps or layers, overlooking the evolving semantic focus across denoising stages and Transformer blocks.To address this, we propose Sortblock, a training-free inference acceleration framework that dynamically caches block-wise features based on their similarity across adjacent timesteps. By ranking the evolution of residuals, Sortblock adaptively determines a recomputation ratio, selectively skipping redundant computations while preserving generation quality. Furthermore, we incorporate a lightweight linear prediction mechanism to reduce accumulated errors in skipped blocks.Extensive experiments across various tasks and DiT architectures demonstrate that Sortblock achieves over 2$\times$ inference speedup with minimal degradation in output quality, offering an effective and generalizable solution for accelerating diffusion-based generative models.

Figures (8)

• Figure 1: Visualization of output differences in consecutive timesteps of Wan2.1, HunyuanVideo and Flux.1-dev.
• Figure 2: Visualization of Flux.1-dev blocks. (a) Visualization of block dynamics as the network depth and timesteps increase. (b) L1 distance variations between dual-stream attention blocks in Flux.1-dev. (c) L1 distance variations between single-stream attention blocks in Flux.1-dev.
• Figure 3: The SortBlock pipeline. Block features are adaptively updated based on their similarity between fully computed and predicted counterparts. Specifically, blocks with low similarity are recomputed, while the others are efficiently approximated via prediction.
• Figure 4: Correlation visualization between input and output differences across consecutive timesteps in Wan2.1, Flux.1-dev and HunyuanVideo. While raw data points deviate significantly from linearity, polynomial fitting improves the approximation accuracy.
• Figure 5: Quality-latency comparison of cache method with different speed. PSNR and SSIM comparisons of SortBlock, T-GATE, PAB, TaylorSeer, and TeaCache using Flux.1-dev. SortBlock achieves the best overall balance between visual quality and efficiency.