Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation

TL;DR

Causal Forcing is proposed that uses an AR teacher for ODE initialization, thereby bridges the architectural gap and outperforms all baselines across all metrics, surpassing the SOTA Self Forcing.

Abstract

To achieve real-time interactive video generation, current methods distill pretrained bidirectional video diffusion models into few-step autoregressive (AR) models, facing an architectural gap when full attention is replaced by causal attention. However, existing approaches do not bridge this gap theoretically. They initialize the AR student via ODE distillation, which requires frame-level injectivity, where each noisy frame must map to a unique clean frame under the PF-ODE of an AR teacher. Distilling an AR student from a bidirectional teacher violates this condition, preventing recovery of the teacher's flow map and instead inducing a conditional-expectation solution, which degrades performance. To address this issue, we propose Causal Forcing that uses an AR teacher for ODE initialization, thereby bridging the architectural gap. Empirical results show that our method outperforms all baselines across all metrics, surpassing the SOTA Self Forcing by 19.3\% in Dynamic Degree, 8.7\% in VisionReward, and 16.7\% in Instruction Following. Project page and the code: \href{https://thu-ml.github.io/CausalForcing.github.io/}{https://thu-ml.github.io/CausalForcing.github.io/}

Figures (10)

• Figure 1: Limitations of existing methods. While distilling from the same bidirectional base model, SOTA autoregressive diffusion distillation methods like Self-Forcing still lag significantly behind standard DMD, which distills a bidirectional student.
• Figure 2: DMD fails to bridge the architectural gap. Initializing the autoregressive student with standard DMD removes the sampling-step gap and isolates the architectural gap, yet still underperforms standard DMD. This indicates that the architectural gap cannot be resolved by the DMD stage and should instead be addressed during the preceding ODE initialization.
• Figure 3: Necessary principle for ODE initialization and why Self Forcing is flawed. ODE distillation requires injective paired data. (a) Standard ODE distillation, which distills a bidirectional teacher to a bidirectional student, satisfies this requirement at the video level. (b) For an AR student, injectivity must hold at the frame level: each noisy frame maps to a unique clean frame via the PF-ODE of the AR teacher. (c) In contrast, Self Forcing distills an AR student from a bidirectional teacher, where the same noisy frame corresponds to multiple distinct clean frames, violating frame-level injectivity and results in blurred videos after ODE distillation. See Sec. \ref{['sec:analysis']} for details.
• Figure 4: TF vs. DF in AR diffusion training. Contrary to common belief, DF leads to video collapse due to the training-inference gap, whereas TF produces higher visual quality.
• Figure 5: Performance comparison between Self Forcing (SF) and ours. DMD with Self Forcing's ODE initialization shows weaker dynamics and artifacts, whereas with causal ODE initialization, it achieves stronger dynamics with higher visual fidelity.

Theorems & Definitions (9)

• Definition 3.1: Frame-level injectivity
• Lemma 3.2: Frame-level non-injectivity of PF-ODE, informal
• Proposition 3.3: Distribution mismatch in current Self Forcing ODE distillation, proof in Appendix \ref{['sec:appendix_proof_asy_ode_is_wrong']}
• Proposition 3.4: Distribution mismatch in autoregressive diffusion forcing
• Lemma 2.1: Chunk-wise non-injectivity of PF-ODE
• proof
• Proposition 2.2: Distribution mismatch in chunk-wise regression
• proof
• proof