Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusion

TL;DR

PaFu-KV tackles memory and latency challenges in autoregressive video diffusion by introducing a past- and future-informed KV Cache policy. It leverages a lightweight Salience Estimation Head trained via Distribution Matching Distillation from a bidirectional teacher to estimate per-token salience, guiding eviction of low-utility KV entries. During inference, salient tokens are retained to maintain fidelity with a compact KV Cache, enabling longer-horizon generation with improved temporal coherence. Empirical results on short and long video benchmarks show faster inference and strong temporal consistency with minimal quality loss, supported by targeted ablations of SEH design and salience scoring.

Abstract

Video generation is pivotal to digital media creation, and recent advances in autoregressive video generation have markedly enhanced the efficiency of real-time video synthesis. However, existing approaches generally rely on heuristic KV Cache policies, which ignore differences in token importance in long-term video generation. This leads to the loss of critical spatiotemporal information and the accumulation of redundant, invalid cache, thereby degrading video generation quality and efficiency. To address this limitation, we first observe that token contributions to video generation are highly time-heterogeneous and accordingly propose a novel Past- and Future-Informed KV Cache Policy (PaFu-KV). Specifically, PaFu-KV introduces a lightweight Salience Estimation Head distilled from a bidirectional teacher to estimate salience scores, allowing the KV cache to retain informative tokens while discarding less relevant ones. This policy yields a better quality-efficiency trade-off by shrinking KV cache capacity and reducing memory footprint at inference time. Extensive experiments on benchmarks demonstrate that our method preserves high-fidelity video generation quality while enables accelerated inference, thereby enabling more efficient long-horizon video generation. Our code will be released upon paper acceptance.

Figures (8)

• Figure 1: By distilling both past and future contextual information from a bidirectional teacher model, our PaFu-KV retaining KV Cache token with high salience score, achieving less KV Cache size and faster inference with negligible degradation on generation quality.
• Figure 2: Overall framework of PaFu-KV. (a) Overview of the salience score criterion used during training. (b) Detailed training procedure of PaFu-KV. The salience score list is maintained in sync with the KV Cache. (c) Inference pipeline of PaFu-KV, where the KV Cache is maintained at a compact size via salience-based eviction.
• Figure 3: (a) A $7 \times 7$ count matrix constructed by uniformly partitioning key indices with maximal query responses and the token index range. (b) Overlap ratio of top-$k$ salient token indices between each intermediate Transformer layer and the final layer of Wan2.1-14B at the final denoising step $t_0$.
• Figure 4: Qualitative experimental results on 30-second videos. We compare PaFu-KV with representative open-source autoregressive video generation models. We explicitly mark the inconsistent regions using a red circle in the figure for better visualization.
• Figure 5: Qualitative experimental results on 30-second videos. We compare PaFu-KV with representative open-source autoregressive video generation models. We explicitly mark the inconsistent regions using a red circle in the figure for better visualization.