SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation

TL;DR

SpecDiff tackles diffusion-model inference acceleration by rethinking feature caching as an information-utilization problem. It introduces a training-free, self-speculation-based framework that combines limited future information with historical signals to guide token caching, using a novel token-importance score and a three-tier feature-computation scheme. The approach yields substantial end-to-end speedups (up to 3.17×) on multiple DiT-based models with negligible quality loss and improved image-text alignment, outperforming prior caching methods. By merging speculative and historical information, SpecDiff expands the Pareto frontier between speed and accuracy, offering a practical path to faster diffusion in real deployments.

Abstract

Feature caching has recently emerged as a promising method for diffusion model acceleration. It effectively alleviates the inefficiency problem caused by high computational requirements by caching similar features in the inference process of the diffusion model. In this paper, we analyze existing feature caching methods from the perspective of information utilization, and point out that relying solely on historical information will lead to constrained accuracy and speed performance. And we propose a novel paradigm that introduces future information via self-speculation based on the information similarity at the same time step across different iteration times. Based on this paradigm, we present \textit{SpecDiff}, a training-free multi-level feature caching strategy including a cached feature selection algorithm and a multi-level feature classification algorithm. (1) Feature selection algorithm based on self-speculative information. \textit{SpecDiff} determines a dynamic importance score for each token based on self-speculative information and historical information, and performs cached feature selection through the importance score. (2) Multi-level feature classification algorithm based on feature importance scores. \textit{SpecDiff} classifies tokens by leveraging the differences in feature importance scores and introduces a multi-level feature calculation strategy. Extensive experiments show that \textit{SpecDiff} achieves average 2.80 \times, 2.74 \times , and 3.17\times speedup with negligible quality loss in Stable Diffusion 3, 3.5, and FLUX compared to RFlow on NVIDIA A800-80GB GPU. By merging speculative and historical information, \textit{SpecDiff} overcomes the speedup-accuracy trade-off bottleneck, pushing the Pareto frontier of speedup and accuracy in the efficient diffusion model inference.

Figures (8)

• Figure 1: Pareto frontier of accuracy and speedup towards DiT feature caching. The detailed normalized accuracy and speedup are obtained with Stable Diffusion 3 and FLUX on an NVIDIA A800-80GB GPU. CR represents the caching ratio in the configuration of feature caching methods.
• Figure 2: Overview of SpecDiff. (a) Dataflow of Diffusion Transformer with feature caching. (b) Key insight: Highly similar information from the same time step in different iterations can serve as future information. (c) Two main techniques of SpecDiff. (d) Results on accuracy and speedup.
• Figure 3: Analysis on existing feature caching methods from the perspective of information utilization.
• Figure 4: (a) The recall of selected tokens is defined as the proportion of tokens that are actually important in the time step being successfully predicted. When we add future information, we can get closer to achieving the theoretical upper limit of prediction. (b) Tokens that cannot be obtained by historical information will appear in future information. And the recall is much higher than the random method. (c) SpecDiff uses a few-step inference to obtain future information for token selection.
• Figure 5: (a) Without starvation scores, the distribution of tokens selected appears skewed. (b) Higher score tokens appear to have a higher ERROR coefficient of variation than lower ones. (c) Noise feature appears high similarity in continuous 3 steps.