Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Parallel Diffusion Model Serving with Residual Compression

Jiajun Luo, Yicheng Xiao, Jianru Xu, Yangxiu You, Rongwei Lu, Chen Tang, Jingyan Jiang, Zhi Wang

TL;DR

CompactFusion tackles the communication bottleneck in parallel diffusion serving by introducing Residual Compression, which transmits compressed activation residuals with an error-feedback loop to prevent drift. By leveraging temporal redundancy, it eliminates redundant data rather than merely overlapping communication with computation, enabling aggressive compression (including 1–100x regimes) without sacrificing fidelity. The approach is modular, hardware- and framework-agnostic, and demonstrates substantial speedups (up to 3x–6.7x across various networks) while maintaining high-quality image and video generation. Its lightweight integration, cross-framework applicability, and support for aggressive compression make it a practical paradigm shift for diffusion inference at scale.

Abstract

Diffusion models produce realistic images and videos but require substantial computational resources, necessitating multi-accelerator parallelism for real-time deployment. However, parallel inference introduces significant communication overhead from exchanging large activations between devices, limiting efficiency and scalability. We present CompactFusion, a compression framework that significantly reduces communication while preserving generation quality. Our key observation is that diffusion activations exhibit strong temporal redundancy-adjacent steps produce highly similar activations, saturating bandwidth with near-duplicate data carrying little new information. To address this inefficiency, we seek a more compact representation that encodes only the essential information. CompactFusion achieves this via Residual Compression that transmits only compressed residuals (step-wise activation differences). Based on empirical analysis and theoretical justification, we show that it effectively removes redundant data, enabling substantial data reduction while maintaining high fidelity. We also integrate lightweight error feedback to prevent error accumulation. CompactFusion establishes a new paradigm for parallel diffusion inference, delivering lower latency and significantly higher generation quality than prior methods. On 4xL20, it achieves 3.0x speedup while greatly improving fidelity. It also uniquely supports communication-heavy strategies like sequence parallelism on slow networks, achieving 6.7x speedup over prior overlap-based method. CompactFusion applies broadly across diffusion models and parallel settings, and integrates easily without requiring pipeline rework. Portable implementation demonstrated on xDiT is publicly available at https://github.com/Cobalt-27/CompactFusion

Accelerating Parallel Diffusion Model Serving with Residual Compression

TL;DR

CompactFusion tackles the communication bottleneck in parallel diffusion serving by introducing Residual Compression, which transmits compressed activation residuals with an error-feedback loop to prevent drift. By leveraging temporal redundancy, it eliminates redundant data rather than merely overlapping communication with computation, enabling aggressive compression (including 1–100x regimes) without sacrificing fidelity. The approach is modular, hardware- and framework-agnostic, and demonstrates substantial speedups (up to 3x–6.7x across various networks) while maintaining high-quality image and video generation. Its lightweight integration, cross-framework applicability, and support for aggressive compression make it a practical paradigm shift for diffusion inference at scale.

Abstract

Diffusion models produce realistic images and videos but require substantial computational resources, necessitating multi-accelerator parallelism for real-time deployment. However, parallel inference introduces significant communication overhead from exchanging large activations between devices, limiting efficiency and scalability. We present CompactFusion, a compression framework that significantly reduces communication while preserving generation quality. Our key observation is that diffusion activations exhibit strong temporal redundancy-adjacent steps produce highly similar activations, saturating bandwidth with near-duplicate data carrying little new information. To address this inefficiency, we seek a more compact representation that encodes only the essential information. CompactFusion achieves this via Residual Compression that transmits only compressed residuals (step-wise activation differences). Based on empirical analysis and theoretical justification, we show that it effectively removes redundant data, enabling substantial data reduction while maintaining high fidelity. We also integrate lightweight error feedback to prevent error accumulation. CompactFusion establishes a new paradigm for parallel diffusion inference, delivering lower latency and significantly higher generation quality than prior methods. On 4xL20, it achieves 3.0x speedup while greatly improving fidelity. It also uniquely supports communication-heavy strategies like sequence parallelism on slow networks, achieving 6.7x speedup over prior overlap-based method. CompactFusion applies broadly across diffusion models and parallel settings, and integrates easily without requiring pipeline rework. Portable implementation demonstrated on xDiT is publicly available at https://github.com/Cobalt-27/CompactFusion

Paper Structure

This paper contains 64 sections, 3 theorems, 31 equations, 10 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries on Parallelism
  3. Method
  4. Exploiting Temporal Redundancy via Residual Compression
  5. Compensating Error Accumulation with Feedback
  6. Scaling Residual Compression to Extreme Ratios
  7. System Co-Design for Efficiency and Usability
  8. Optimized Compression Kernels.
  9. Latency Hiding via Overlap.
  10. Ease of Integration.
  11. Experiments
  12. Setups
  13. Models.
  14. Dataset.
  15. Hardware.
  16. ...and 49 more sections

Key Result

Proposition 3.1

Let $v^{\text{naive}}$ and $v^{\text{residual}}$ denote the steady-state mean squared error upper bounds under naive compression and residual compression with error feedback, respectively. Then under the given assumptions, their ratio satisfies the bound

Figures (10)

  • Figure 1: CompactFusion is a compression framework for parallel diffusion serving acceleration. Leveraging the intrinsic redundancy in diffusion models, CompactFusion transmits compressed step-wise residuals instead of full activation to significantly reduce communication volume and preserve fidelity, while prior works rely on stale activations for computation-communication overlap, leading to noticeable degradation. Setting: 4$\times$L20, FLUX-1.dev, 28-step, 1024$\times$1024 resolution, 1-step warmup for all algorithms.
  • Figure 2: We introduce Residual Compression as a new paradigm for efficient parallel diffusion. It generalizes across compressors and sustains high quality even under $100.05\times$ compression. Setups: 4 devices, 1-step warmup.
  • Figure 3: (a) CompactFusion prevents long-term error accumulation by adding feedback into residual compression. (b) By wrapping a few standard communication primitives with a unified compression interface, CompactFusion requires minimal change to the existing frameworks and can easily be integrated to new model structure.
  • Figure 4: Comparison of strategies under 1-bit quantization: original (no compression), naive compression, residual compression with/without error feedback. Left: Visual Results, right: Error Analysis and PCA Trajectories (closer better) of activations over steps. We report two metrics: Compression Error (difference between input and output of compression) and Total Error (cumulative deviation from the uncompressed ground truth, lower is better). Setups: 4 devices, 1-step warmup.
  • Figure 5: Adjacent-Step Activation Similarity and Activation/Residual Norm, on Flux.
  • ...and 5 more figures

Theorems & Definitions (3)

  • Proposition 3.1: Steady-State Error Bound
  • Proposition H.1
  • Proposition H.2