Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation

Yang Xiao, Gen Li, Kaiyuan Deng, Yushu Wu, Zheng Zhan, Yanzhi Wang, Xiaolong Ma, Bo Hui

TL;DR

LightCache tackles the memory bottleneck in diffusion-based video generation by analyzing stage-wise computation (encoding, denoising, decoding) and introducing three training-free strategies: Asynchronous cache swapping, feature chunking, and VAE slicing for decoding. These stage-specific optimizations reduce peak GPU memory while keeping latency overhead below the acceleration gains, outperforming prior training-free and caching baselines like DeepCache and FME. Ablation studies confirm each component contributes to memory reduction, and the Euler scheduler yields the best balance of speed and quality. The approach enables faster, more memory-efficient video generation on multi-GPU setups, with practical applicability to DiT-based architectures and longer sequences.

Abstract

Training-free acceleration has emerged as an advanced research area in video generation based on diffusion models. The redundancy of latents in diffusion model inference provides a natural entry point for acceleration. In this paper, we decompose the inference process into the encoding, denoising, and decoding stages, and observe that cache-based acceleration methods often lead to substantial memory surges in the latter two stages. To address this problem, we analyze the characteristics of inference across different stages and propose stage-specific strategies for reducing memory consumption: 1) Asynchronous Cache Swapping. 2) Feature chunk. 3) Slicing latents to decode. At the same time, we ensure that the time overhead introduced by these three strategies remains lower than the acceleration gains themselves. Compared with the baseline, our approach achieves faster inference speed and lower memory usage, while maintaining quality degradation within an acceptable range. The Code is available at https://github.com/NKUShaw/LightCache .

LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation

TL;DR

LightCache tackles the memory bottleneck in diffusion-based video generation by analyzing stage-wise computation (encoding, denoising, decoding) and introducing three training-free strategies: Asynchronous cache swapping, feature chunking, and VAE slicing for decoding. These stage-specific optimizations reduce peak GPU memory while keeping latency overhead below the acceleration gains, outperforming prior training-free and caching baselines like DeepCache and FME. Ablation studies confirm each component contributes to memory reduction, and the Euler scheduler yields the best balance of speed and quality. The approach enables faster, more memory-efficient video generation on multi-GPU setups, with practical applicability to DiT-based architectures and longer sequences.

Abstract

Training-free acceleration has emerged as an advanced research area in video generation based on diffusion models. The redundancy of latents in diffusion model inference provides a natural entry point for acceleration. In this paper, we decompose the inference process into the encoding, denoising, and decoding stages, and observe that cache-based acceleration methods often lead to substantial memory surges in the latter two stages. To address this problem, we analyze the characteristics of inference across different stages and propose stage-specific strategies for reducing memory consumption: 1) Asynchronous Cache Swapping. 2) Feature chunk. 3) Slicing latents to decode. At the same time, we ensure that the time overhead introduced by these three strategies remains lower than the acceleration gains themselves. Compared with the baseline, our approach achieves faster inference speed and lower memory usage, while maintaining quality degradation within an acceptable range. The Code is available at https://github.com/NKUShaw/LightCache .

Paper Structure

This paper contains 22 sections, 9 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminary
  4. Diffusion Forward Process.
  5. Diffusion Reverse Process.
  6. Feature Calculation
  7. Asynchronous Swapping
  8. Feature chunk and Sliced Decoding
  9. Experiments
  10. Analysis
  11. Ablation Studies
  12. Three Modules
  13. Scheduler
  14. Visualization
  15. Related Works
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Accelerating AnimateDiff-Lightning and Stable-Video-Diffusion-Img2vid-XT by 1.59× and 2.86×, while reducing memory usage by 8.0 GB and 1.4 GB, respectively.
  • Figure 2: Highly similar feature map between all up-sampling and down-sampling layers
  • Figure 3: Memory Usage
  • Figure 4: Denoising: The timestep is divided into a cache step and a normal step. The cache step reuses the cached feature maps stored in the CPU. The normal step chunk the size of feature maps and concatenate them. Decoding: We use VAE to decode the latent frame by frame.
  • Figure 5: Violin Figure (AnimateDiff-Light) of Quality Metric: LPIPS(↓), PSNR(↑), SSIM(↑)
  • ...and 2 more figures