PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache

TL;DR

The paper tackles the KV-cache bottleneck in unified autoregressive video generation, where per-step attention costs scale with sequence length and the cache grows linearly with frames. It introduces PackCache, a training-free KV-cache management method that preserves semantic anchors (text prompts and conditioning images) and allocates a fixed, decayed budget across past frames using an exponential kernel, complemented by Spatially Preserving 3D rotary position embeddings to maintain spatiotemporal coherence. The approach is validated on the Lumos-1 model across 24- and 48-frame sequences on A40 and H200 GPUs, achieving 1.7–2.2× end-to-end speedups and up to 2.6–3.7× acceleration for the final, most expensive frames, while preserving quality on key I2VBench metrics. These results demonstrate that training-free cache compaction can enable longer-horizon, high-quality video generation within fixed memory budgets, significantly expanding the practical reach of unified autoregressive video systems.

Abstract

A unified autoregressive model is a Transformer-based framework that addresses diverse multimodal tasks (e.g., text, image, video) as a single sequence modeling problem under a shared token space. Such models rely on the KV-cache mechanism to reduce attention computation from O(T^2) to O(T); however, KV-cache size grows linearly with the number of generated tokens, and it rapidly becomes the dominant bottleneck limiting inference efficiency and generative length. Unified autoregressive video generation inherits this limitation. Our analysis reveals that KV-cache tokens exhibit distinct spatiotemporal properties: (i) text and conditioning-image tokens act as persistent semantic anchors that consistently receive high attention, and (ii) attention to previous frames naturally decays with temporal distance. Leveraging these observations, we introduce PackCache, a training-free KV-cache management method that dynamically compacts the KV cache through three coordinated mechanisms: condition anchoring that preserves semantic references, cross-frame decay modeling that allocates cache budget according to temporal distance, and spatially preserving position embedding that maintains coherent 3D structure under cache removal. In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences, showcasing its strong potential for enabling longer-sequence video generation. Notably, the final four frames - the portion most impacted by the progressively expanding KV-cache and thus the most expensive segment of the clip - PackCache delivers a 2.6x and 3.7x acceleration on A40 and H200, respectively, for 48-frame videos.

Figures (13)

• Figure 1: 24-Frame generation overhead of unified autoregressive video models on an NVIDIA A40 GPU. (a) Cumulative generation time for a 24-frame video ($384\times672$) using the 3B-parameter Lumos-1. (b) Per-frame generation time. The cumulative curve exhibits rapidly increasing trend as the sequence length increases, and the incremental cost of later frames rises to over $160$ seconds per frame, indicating severe KV-cache–driven scaling bottlenecks. Our method accelerates end-to-end generation by $1.7\times$ compared to the baseline, and achieves a $2.6\times$ speedup on the final four frames when generating 48-frame videos on the A40 GPU.
• Figure 2: DecoderLayer Bottleneck Comparison Relative time distribution of module-level operations in three decoder layers. Attention dominates computation (71-76%), with LayerNorm and QK_Norm contributing 9% each. Values represent percentage of total layer execution time, measured with GPU synchronization over 612 forward passes.
• Figure 3: Attention heatmap and attention-distribution visualizations of the 1B Lumos-1 model during the generation of latent frame 3. In the heatmaps (a,d), the horizontal axis enumerates KV-cache key tokens: $[0,249)$ corresponds to text-prompt tokens, $[249,2077)$ to the conditioning image, $[2077,3905)$ to the earliest cached frame, $[3905,5733)$ to the second-most recent frame, and $[5733,7561)$ to the masked current-frame tokens. The vertical axis represents the KV-cache query positions of the current frame. Each heatmap shows a single timestep from a selected layer, averaged over all attention heads.Autoregressive generation proceeds by iteratively predicting masked tokens over multiple timesteps. The attention-distribution plots (b,c,e,f) aggregate the mean attention scores (averaged across heads) from current-frame queries to each semantic region in the cache. These visualizations reveal three consistent patterns: (1) a rightward shift in attention toward more recent frames, (2) a monotonic temporal decay across history frames, and (3) strong, persistent attention to the text prompt and conditioning image, which act as stable semantic anchors throughout autoregressive video synthesis.
• Figure 4: The AR-DF pipeline in Lumos-1yuan2025lumos. In the top row, video frames are generated autoregressively, where each frame is conditioned on text prompts and partial visual tokens from previous frames stored in the KV cache. The text prompt and condition image are encoded and stored in the KV cache as global context. The partially visible tokens are written into the KV cache as contextual information for subsequent frames. In the bottom row, each frame is synthesized via discrete diffusion, progressively replacing masked tokens with valid visual tokens until a complete frame is obtained.
• Figure 5: Comparison of KV-cache strategies in unified autoregressive video generation. At each timestep, the model predicts the current masked frame by attending to the text prompt, conditioning image, and cached previous frames. Once all tokens of the current frame are predicted, the resulting latent frame is written into the KV cache. (a) Full KV-Cache stores all previously generated frames, enabling complete cache reuse but resulting in a KV-cache that grows linearly with video length $T$, leading to quadratic attention cost. (b) Sliding KV-Cache keeps only the most recent $W$ frames, reducing complexity to $\mathcal{O}(TW)$ but sliding out older frames and thus losing long-range temporal context. (c) Compact KV-Cache (ours) compacts previous frames based on their spatiotemporal relevance, retaining only informative tokens while staying within the same window budget $W$. This preserves long-range context through effective cache reuse while maintaining the same $\mathcal{O}(TW)$ complexity as the sliding-window approach.