Causality in Video Diffusers is Separable from Denoising

TL;DR

The paper addresses the challenge of modeling temporal causality in video diffusion by showing that causal reasoning is separable from the multi-step denoising process. It identifies two regularities—redundant early denoiser computations across steps and sparse cross-frame attention in deeper layers—and proposes Separable Causal Diffusion (SCD), which decouples once-per-frame causal reasoning into a Temporal Causal Encoder from a lightweight Frame-wise Diffusion Decoder. SCD achieves substantial throughput and per-frame latency reductions while matching or surpassing the generation quality of strong causal baselines, demonstrated across pretraining and post-training tasks on synthetic and real datasets. The results suggest practical benefits for real-time video generation and streaming applications and highlight avenues for scalable deployment by separating temporal reasoning from frame-wise rendering.

Abstract

Causality -- referring to temporal, uni-directional cause-effect relationships between components -- underlies many complex generative processes, including videos, language, and robot trajectories. Current causal diffusion models entangle temporal reasoning with iterative denoising, applying causal attention across all layers, at every denoising step, and over the entire context. In this paper, we show that the causal reasoning in these models is separable from the multi-step denoising process. Through systematic probing of autoregressive video diffusers, we uncover two key regularities: (1) early layers produce highly similar features across denoising steps, indicating redundant computation along the diffusion trajectory; and (2) deeper layers exhibit sparse cross-frame attention and primarily perform intra-frame rendering. Motivated by these findings, we introduce Separable Causal Diffusion (SCD), a new architecture that explicitly decouples once-per-frame temporal reasoning, via a causal transformer encoder, from multi-step frame-wise rendering, via a lightweight diffusion decoder. Extensive experiments on both pretraining and post-training tasks across synthetic and real benchmarks show that SCD significantly improves throughput and per-frame latency while matching or surpassing the generation quality of strong causal diffusion baselines.

Figures (14)

• Figure 1: Causality in autoregressive video diffusion models is separable from the denoising process. The prevailing design of causal diffusion models for visual generation performs causal attention densely across all layers and all denoising steps (left). However, we uncover two important observations (right): 1) early denoiser layers share highly repetitive computation across denoising steps (blue); 2) deep layers primarily attend to intra-frame tokens, with sparse cross-frame connections (red).
• Figure 2: Strong middle-block feature consistency across denoising steps. (a) When denoising the same frame over 50 steps, the middle-block (15th block out of 30) features exhibit consistently high cosine similarity (above 0.95), suggesting that the features generated in the middle block are mostly shared across different diffusion steps. (b) PCA analysis further confirms that the middle-block features at the first and later diffusion steps are highly aligned, indicating that structures are effectively established even in the first step.
• Figure 3: Skipping the middle layers across denoising steps. To take advantage of the repetitive computation, we finetune with a skip-layer design: except for the starting denoising steps, the denoiser skips a large chunk of 15 (out of 30) middle layers during diffusion. After short finetuning, semantics, layout, and motion are preserved and visual fidelity is restored. Full details on the design of this finetuning are provided in Appendix \ref{['sec:app-obs-redundancy']}.
• Figure 4: Cross‑frame attention becomes sparse with depth. For a newly denoised frame $i$, we aggregate, for each transformer layer and attention head, the attention mass that query tokens at $i$ assign to keys from its context frames. Results indicate that deeper layers allocate markedly less mass to past frames, indicating they focus on intra‑frame refinement, and cross-frame attention is largely unnecessary.
• Figure 5: Removing deep cross‑frame attention. We switch the last $5$ (of $30$) layers from a frame‑causal mask to a frame‑diagonal mask, removing their access to context‑frame KV caches. A brief $5$k‑step finetune with the frame-diagonal mask stabilizes the semantics, layout, and motion and restores visual fidelity.