Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization

TL;DR

This work tackles the memory bottleneck of KV-cache in autoregressive video diffusion by introducing Quant VideoGen (QVG), a training-free KV-cache quantization framework that leverages video-specific spatiotemporal redundancy. It combines Semantic-Aware Smoothing to form semantically similar token groups and Progressive Residual Quantization to encode residuals in multiple coarse-to-fine stages, with efficient algorithm-system co-design for streaming inference. Across multiple models and benchmarks, QVG achieves up to $7.0\times$ KV-cache compression with less than $4\%$ end-to-end latency overhead while maintaining near-lossless video quality, enabling practical long-horizon generation on commodity hardware. These contributions significantly advance deployability and real-time capabilities for long video generation and world-model applications by reducing memory footprints without sacrificing perceptual fidelity.

Abstract

Despite rapid progress in autoregressive video diffusion, an emerging system algorithm bottleneck limits both deployability and generation capability: KV cache memory. In autoregressive video generation models, the KV cache grows with generation history and quickly dominates GPU memory, often exceeding 30 GB, preventing deployment on widely available hardware. More critically, constrained KV cache budgets restrict the effective working memory, directly degrading long horizon consistency in identity, layout, and motion. To address this challenge, we present Quant VideoGen (QVG), a training free KV cache quantization framework for autoregressive video diffusion models. QVG leverages video spatiotemporal redundancy through Semantic Aware Smoothing, producing low magnitude, quantization friendly residuals. It further introduces Progressive Residual Quantization, a coarse to fine multi stage scheme that reduces quantization error while enabling a smooth quality memory trade off. Across LongCat Video, HY WorldPlay, and Self Forcing benchmarks, QVG establishes a new Pareto frontier between quality and memory efficiency, reducing KV cache memory by up to 7.0 times with less than 4% end to end latency overhead while consistently outperforming existing baselines in generation quality.

Figures (7)

• Figure 1: QVG makes long video generation extremely memory-efficient and maintains high video quality. On LongCat-Video and HY-WorldPlay, QVG reduces the memory footprint by up to $7\times$ and achieves a PSNR of 28.7, much better than the baseline.
• Figure 2: (a) Adopting full KV-cache can resolve the drifting problem but is very likely to be bottlenecked by memory. QVG can successfully generation high-quality long-videos. (b) Video diffusion models exhibit substantial spatiotemporal redundancy: tokens that are spatially or temporally adjacent have high cosine similarity, making compression feasible.
• Figure 3: (a-c) Semantic-Aware Smoothing effectively smoothing the KV-cache distribution to make it more regular and quantization-friendly. We (1) group similar tokens together based on their semantic similarity and (2) subtract the centroid for each group to smooth the distribution. (d) The magnitude is significantly reduced and more concentrated around 0, making it much easier to be quantized.
• Figure 4: Overview of QVG framework. (a) Original tensor's distribution is irregular and hard to quantize. (b) Semantic-Aware Smoothing groups similar tokens and subtracts centroids for each group to make the residual quantization friendly. (c) Progressive Residual Quantization further lowers quantization error by iteratively applying Semantic-Aware Smoothing algorithm. (d) The final residual tensor becomes much easier to quantize and has a much lower quantization error.
• Figure 5: (a–b) Imaging Quality over long-horizon generation on Self-Forcing Model. Both QVG and QVG-Pro preserve near-lossless quality, while prior baselines degrade drastically. (c) The first stage of Progressive Residual Quantization yields the most significant reduction in MSE. Subsequent stages further reduce the error, but with diminishing returns.