Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Plug-and-Play Fidelity Optimization for Diffusion Transformer Acceleration via Cumulative Error Minimization

Tong Shao, Yusen Fu, Guoying Sun, Jingde Kong, Zhuotao Tian, Jingyong Su

TL;DR

Diffusion Transformer models suffer slow inference due to sequential denoising, and fixed caching strategies fail to adapt to complex denoising dynamics. The paper proposes Cumulative Error Minimization (CEM), a training-free plug-in that builds an offline prior of caching error as a function of timestep $t$ and cache interval $n$, then uses dynamic programming to minimize the cumulative error under an acceleration budget $N_c$. CEM is model-agnostic, compatible with existing error-correction and quantized DiTs, and requires no online overhead beyond a precomputed error matrix. Across multiple text-to-image, text-to-video, and class-to-image tasks, CEM consistently improves fidelity for eight generation models and a quantized DiT, while preserving or enhancing acceleration, demonstrating a practical path to robust, high-fidelity, fast diffusion generation.

Abstract

Although Diffusion Transformer (DiT) has emerged as a predominant architecture for image and video generation, its iterative denoising process results in slow inference, which hinders broader applicability and development. Caching-based methods achieve training-free acceleration, while suffering from considerable computational error. Existing methods typically incorporate error correction strategies such as pruning or prediction to mitigate it. However, their fixed caching strategy fails to adapt to the complex error variations during denoising, which limits the full potential of error correction. To tackle this challenge, we propose a novel fidelity-optimization plugin for existing error correction methods via cumulative error minimization, named CEM. CEM predefines the error to characterize the sensitivity of model to acceleration jointly influenced by timesteps and cache intervals. Guided by this prior, we formulate a dynamic programming algorithm with cumulative error approximation for strategy optimization, which achieves the caching error minimization, resulting in a substantial improvement in generation fidelity. CEM is model-agnostic and exhibits strong generalization, which is adaptable to arbitrary acceleration budgets. It can be seamlessly integrated into existing error correction frameworks and quantized models without introducing any additional computational overhead. Extensive experiments conducted on nine generation models and quantized methods across three tasks demonstrate that CEM significantly improves generation fidelity of existing acceleration models, and outperforms the original generation performance on FLUX.1-dev, PixArt-$α$, StableDiffusion1.5 and Hunyuan. The code will be made publicly available.

Plug-and-Play Fidelity Optimization for Diffusion Transformer Acceleration via Cumulative Error Minimization

TL;DR

Diffusion Transformer models suffer slow inference due to sequential denoising, and fixed caching strategies fail to adapt to complex denoising dynamics. The paper proposes Cumulative Error Minimization (CEM), a training-free plug-in that builds an offline prior of caching error as a function of timestep and cache interval , then uses dynamic programming to minimize the cumulative error under an acceleration budget . CEM is model-agnostic, compatible with existing error-correction and quantized DiTs, and requires no online overhead beyond a precomputed error matrix. Across multiple text-to-image, text-to-video, and class-to-image tasks, CEM consistently improves fidelity for eight generation models and a quantized DiT, while preserving or enhancing acceleration, demonstrating a practical path to robust, high-fidelity, fast diffusion generation.

Abstract

Although Diffusion Transformer (DiT) has emerged as a predominant architecture for image and video generation, its iterative denoising process results in slow inference, which hinders broader applicability and development. Caching-based methods achieve training-free acceleration, while suffering from considerable computational error. Existing methods typically incorporate error correction strategies such as pruning or prediction to mitigate it. However, their fixed caching strategy fails to adapt to the complex error variations during denoising, which limits the full potential of error correction. To tackle this challenge, we propose a novel fidelity-optimization plugin for existing error correction methods via cumulative error minimization, named CEM. CEM predefines the error to characterize the sensitivity of model to acceleration jointly influenced by timesteps and cache intervals. Guided by this prior, we formulate a dynamic programming algorithm with cumulative error approximation for strategy optimization, which achieves the caching error minimization, resulting in a substantial improvement in generation fidelity. CEM is model-agnostic and exhibits strong generalization, which is adaptable to arbitrary acceleration budgets. It can be seamlessly integrated into existing error correction frameworks and quantized models without introducing any additional computational overhead. Extensive experiments conducted on nine generation models and quantized methods across three tasks demonstrate that CEM significantly improves generation fidelity of existing acceleration models, and outperforms the original generation performance on FLUX.1-dev, PixArt-, StableDiffusion1.5 and Hunyuan. The code will be made publicly available.
Paper Structure (35 sections, 2 theorems, 10 equations, 14 figures, 17 tables, 1 algorithm)

This paper contains 35 sections, 2 theorems, 10 equations, 14 figures, 17 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Offline Error Modeling
  5. Dynamic Caching Strategy
  6. Plug-and-Play Deployment
  7. Experiment
  8. Experiment Settings
  9. Main Results
  10. Ablation Studies
  11. Conclusion
  12. The Use of Large Language Models
  13. More Details of Offline Error Modeling
  14. Visualization for Error Distribution
  15. Impact of the Source of Offline Samples
  16. ...and 20 more sections

Key Result

Theorem 1

Under the assumption of unified error distribution, the difference $|\mathcal{E}^*(t, n) - \mathcal{E}(t, n)|$ is bounded by: where $\delta \in (0,1)$ is a confidence parameter, and $\epsilon_{\text{var}}$ is a small variance term (empirically small, as per Fig. fig:error1(a)).

Figures (14)

  • Figure 1: Our CEM significantly reduces caching error while maintaining acceleration, thereby improving the generation fidelity of existing acceleration methods. Comprehensive experiments demonstrate the effectiveness and generalization of CEM.
  • Figure 2: Overview of our CEM framework. It first performs Offline Error Modeling to characterize the model's intrinsic sensitivity to caching under different timesteps and cache intervals, forming an offline prior. Guided by this prior, it employs Dynamic Caching Strategy with dynamic programming to determine the optimal caching strategy that minimizes cumulative error and enhances generation fidelity. Finally, CEM supports Plug-and-Play Deployment and can be seamlessly integrated into existing error correction methods and quantized models.
  • Figure 3: Error Analysis.(a). Mean-variance of offline error modeling under different cache intervals. The error variance remains relatively small across various contents and cache intervals. (b). Consistency between offline modeling and actual inference. The error points obtained during inference fall within the prior-modeled distribution, indicating strong consistency between prior modeling and real inference. (c). Offline cumulative error vs. online error. The cumulative error approximation accurately captures the trend of error variation during actual inference.
  • Figure 4: Qualitative visualization comparison on text-to-image generation. We highlight the areas with red dashed boxes to emphasize the comparison. Our CEM achieves higher generation fidelity under the same or higher acceleration efficiency compared with baselines. More experiment results and visualizations are provided in the Appendix. \ref{['app:results']}, \ref{['app:sd15']} and \ref{['app:pixart']}.
  • Figure 5: Qualitative visualization comparison on Hunyuan. Our CEM improves the TaylorSeer for better consistency with the original model. See Appendix. \ref{['app:hunyuan']} for more visualizations.
  • ...and 9 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2