Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention

Dvir Samuel, Issar Tzachor, Matan Levy, Micahel Green, Gal Chechik, Rami Ben-Ari

TL;DR

This work targets the KV-cache bottleneck in autoregressive video diffusion by identifying three redundancy sources: duplicate keys across frames, semantically evolving yet slowly changing Q/K, and cross-attention over long prompts where only a subset of tokens matters. It introduces a training-free unified framework comprising TempCache for temporal KV compression, AnnCA for cross-attention pruning, and AnnSA for self-attention sparsification, all leveraging lightweight ANN at inference. The proposed approach yields up to 5×–10× end-to-end speedups with nearly constant peak GPU memory over long rollouts while preserving high visual fidelity, outperforming prior caching and sparse-attention methods that either slow down or inflate memory with time. The framework is plug-and-play and compatible with existing autoregressive diffusion backbones and video world models, enabling efficient long-horizon video generation and neural game-engine style world modeling.

Abstract

Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines. However, their core attention layers become a major bottleneck at inference time: as generation progresses, the KV cache grows, causing both increasing latency and escalating GPU memory, which in turn restricts usable temporal context and harms long-range consistency. In this work, we study redundancy in autoregressive video diffusion and identify three persistent sources: near-duplicate cached keys across frames, slowly evolving (largely semantic) queries/keys that make many attention computations redundant, and cross-attention over long prompts where only a small subset of tokens matters per frame. Building on these observations, we propose a unified, training-free attention framework for autoregressive diffusion: TempCache compresses the KV cache via temporal correspondence to bound cache growth; AnnCA accelerates cross-attention by selecting frame-relevant prompt tokens using fast approximate nearest neighbor (ANN) matching; and AnnSA sparsifies self-attention by restricting each query to semantically matched keys, also using a lightweight ANN. Together, these modules reduce attention, compute, and memory and are compatible with existing autoregressive diffusion backbones and world models. Experiments demonstrate up to x5--x10 end-to-end speedups while preserving near-identical visual quality and, crucially, maintaining stable throughput and nearly constant peak GPU memory usage over long rollouts, where prior methods progressively slow down and suffer from increasing memory usage.

Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention

TL;DR

This work targets the KV-cache bottleneck in autoregressive video diffusion by identifying three redundancy sources: duplicate keys across frames, semantically evolving yet slowly changing Q/K, and cross-attention over long prompts where only a subset of tokens matters. It introduces a training-free unified framework comprising TempCache for temporal KV compression, AnnCA for cross-attention pruning, and AnnSA for self-attention sparsification, all leveraging lightweight ANN at inference. The proposed approach yields up to 5×–10× end-to-end speedups with nearly constant peak GPU memory over long rollouts while preserving high visual fidelity, outperforming prior caching and sparse-attention methods that either slow down or inflate memory with time. The framework is plug-and-play and compatible with existing autoregressive diffusion backbones and video world models, enabling efficient long-horizon video generation and neural game-engine style world modeling.

Abstract

Autoregressive video diffusion models enable streaming generation, opening the door to long-form synthesis, video world models, and interactive neural game engines. However, their core attention layers become a major bottleneck at inference time: as generation progresses, the KV cache grows, causing both increasing latency and escalating GPU memory, which in turn restricts usable temporal context and harms long-range consistency. In this work, we study redundancy in autoregressive video diffusion and identify three persistent sources: near-duplicate cached keys across frames, slowly evolving (largely semantic) queries/keys that make many attention computations redundant, and cross-attention over long prompts where only a small subset of tokens matters per frame. Building on these observations, we propose a unified, training-free attention framework for autoregressive diffusion: TempCache compresses the KV cache via temporal correspondence to bound cache growth; AnnCA accelerates cross-attention by selecting frame-relevant prompt tokens using fast approximate nearest neighbor (ANN) matching; and AnnSA sparsifies self-attention by restricting each query to semantically matched keys, also using a lightweight ANN. Together, these modules reduce attention, compute, and memory and are compatible with existing autoregressive diffusion backbones and world models. Experiments demonstrate up to x5--x10 end-to-end speedups while preserving near-identical visual quality and, crucially, maintaining stable throughput and nearly constant peak GPU memory usage over long rollouts, where prior methods progressively slow down and suffer from increasing memory usage.
Paper Structure (25 sections, 2 theorems, 13 equations, 10 figures, 4 tables)

This paper contains 25 sections, 2 theorems, 13 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Auto-regressive Video Diffusion Models
  4. Motivation
  5. Method
  6. Attention as Approximate Nearest Neighbor Search
  7. KV Cache Compression using Temporal Correspondence
  8. Cross-Attention Redundancy
  9. Self-Attention Redundancy
  10. Experiments
  11. Setup
  12. Results
  13. Quantitative Results
  14. Qualitative Results
  15. Summary
  16. ...and 10 more sections

Key Result

Lemma 5.1

Let $q \in \mathbb{R}^{d_k}$ be a query, and let $K = (k_1, \dots, k_n)^\top$ and $V = (v_1, \dots, v_n)^\top$ denote keys and values. Suppose the indices are partitioned into groups $\{G_t\}_{t=1}^g$ such that all keys within a group are identical. Then the standard attention over $(K,V)$ is exactl

Figures (10)

  • Figure 1: Our method substantially accelerates pre-trained autoregressive video diffusion models and autoregressive world models while maintaining high visual quality, by introducing a new KV-cache compression with self- and cross-attention sparsification. On a single H100 GPU, it achieves $5\times$--$10\times$ speedups for multi-minute video generation, without further training/optimization, and keeps peak GPU memory nearly constant over long rollouts.
  • Figure 2: Attention sparsity in autoregressive video diffusion. Attention recall vs. density on Rolling-Forcing liu2025rolling, averaged over transformer blocks (shaded: std). Density is induced by keeping only the highest-attention entries. This achieves high recall, e.g., $\approx$85% at 30% density, indicating substantial sparsity.
  • Figure 3: Semantic structure and temporal redundancy in self-attention features. PCA of self-attention queries $Q$ and cached keys $\hat{K}$ across frames (similar colors denote nearby embeddings) for a generated video of a cat walking toward the camera. The features exhibit semantic clustering (foreground vs. background) and strong key repetition across frames, motivating KV-cache compression.
  • Figure 4: Cross-attention is frame-selective. Input prompt (top) with Per-token cross-attention maps (bottom) for "cat", "van", and "dog" across frames. Attention concentrates on the tokens relevant to the current content (cat early, van during occlusion, dog after transformation), suggesting that pruning irrelevant prompt tokens per frame can reduce cross-attention compute.
  • Figure 5: Temporal correspondence for KV compression. Many key features persist across frames and can be matched using temporal correspondence (colored dots/arrows). Correspondences are recovered by selecting, for each current-frame query, the most related key in a previous frame difftrack
  • ...and 5 more figures

Theorems & Definitions (3)

  • Lemma 5.1: Redundancy-free attention
  • Lemma 1.1: Redundancy-free attention
  • proof