Block Cascading: Training Free Acceleration of Block-Causal Video Models

TL;DR

Block Cascading addresses the speed-quality bottleneck in diffusion-based block-causal video generation by enabling training-free temporal parallelism. It starts future blocks with partially denoised context from predecessors and shares KV across blocks to run multiple denoising steps in parallel on multiple GPUs, removing KV-recaching overhead during interactive switches. The method achieves approximately $2\times$ streaming FPS across model scales (e.g., 1.3B from 16 to 30 FPS; 14B from 4.5 to 12.5 FPS) while preserving video quality according to VBench and user studies. This approach obviates retraining, broadening real-time, controllable diffusion-based video generation for large-scale models without retraining.

Abstract

Block-causal video generation faces a stark speed-quality trade-off: small 1.3B models manage only 16 FPS while large 14B models crawl at 4.5 FPS, forcing users to choose between responsiveness and quality. Block Cascading significantly mitigates this trade-off through training-free parallelization. Our key insight: future video blocks do not need fully denoised current blocks to begin generation. By starting block generation with partially denoised context from predecessors, we transform sequential pipelines into parallel cascades where multiple blocks denoise simultaneously. With 5 GPUs exploiting temporal parallelism, we achieve ~2x acceleration across all model scales: 1.3B models accelerate from 16 to 30 FPS, 14B models from 4.5 to 12.5 FPS. Beyond inference speed, Block Cascading eliminates overhead from KV-recaching (of ~200ms) during context switches for interactive generation. Extensive evaluations validated against multiple block-causal pipelines demonstrate no significant loss in generation quality when switching from block-causal to Block Cascading pipelines for inference. Project Page: https://hmrishavbandy.github.io/block_cascading_page/

Figures (7)

• Figure 1: Block Cascading Inference: Our inference pipeline reduces dependency on clean, denoised blocks for future block generation. For example in here (center), we denoise block $B^1_{t_3}{=}\{f^3,f^4, f^5\}$ and $B^0_{t_2} {=}\{f^3,f^4, f^5\}$ jointly using bidirectional attention instead of waiting for $B^0_{t_c}$ to denoise $B^1_{t_3}$ (left) causvidhuang2025selfyang2025longlive. By reducing dependency on previous blocks, we can free up the inference pipeline and allow parallel processing of multiple blocks (right) to improve generation speed significantly.
• Figure 2: Expensive KV-recaching: KV-recaching can result in FPS drop in interactive generation as previous generated and stored cache has to be recached using new context. We skip KV-recaching as our KV is auto-recomputed using new context.
• Figure 3: Qualitative comparisons: Comparing our bidirectional inference pipeline with corresponding original inference pipelines
• Figure 4: Instantaneous FPS: We measure FPS as time taken to denoise a particular block of 3 latent frames (12 video frames). FPS fluctuates from changing attention window sizes during block-causal denoising and number of parallel blocks in Block Cascading.
• Figure 5: Bidirectional Inference: Causal attention in fully parallelised generation ($\mathcal{P}_4$) can yield artifacts. These can be fixed by using bidirectional attention with same fully parallelised pipeline.