Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QVGen: Pushing the Limit of Quantized Video Generative Models

Yushi Huang, Ruihao Gong, Jing Liu, Yifu Ding, Chengtao Lv, Haotong Qin, Jun Zhang

TL;DR

QVGen addresses the challenge of deploying ultra‑low‑bit video diffusion models by introducing a quantization‑aware training framework that uses learnable auxiliary modules Φ to stabilize convergence. A theoretical regret analysis shows that minimizing the gradient norm ||g_t||_2 is key for QAT stability, which Φ helps achieve. To avoid inference overhead, QVGen employs a rank‑decay schedule that progressively removes Φ via SVD and a rank‑based regularization, yielding a standard 4‑bit quantized model with minimal performance loss. Extensive experiments across CogVideoX and Wan families demonstrate state‑of‑the‑art 3‑bit performance and full‑precision parity at 4‑bit, with substantial memory savings and practical efficiency gains for real‑world deployment.

Abstract

Video diffusion models (DMs) have enabled high-quality video synthesis. Yet, their substantial computational and memory demands pose serious challenges to real-world deployment, even on high-end GPUs. As a commonly adopted solution, quantization has proven notable success in reducing cost for image DMs, while its direct application to video DMs remains ineffective. In this paper, we present QVGen, a novel quantization-aware training (QAT) framework tailored for high-performance and inference-efficient video DMs under extremely low-bit quantization (e.g., 4-bit or below). We begin with a theoretical analysis demonstrating that reducing the gradient norm is essential to facilitate convergence for QAT. To this end, we introduce auxiliary modules ($Φ$) to mitigate large quantization errors, leading to significantly enhanced convergence. To eliminate the inference overhead of $Φ$, we propose a rank-decay strategy that progressively eliminates $Φ$. Specifically, we repeatedly employ singular value decomposition (SVD) and a proposed rank-based regularization $\mathbfγ$ to identify and decay low-contributing components. This strategy retains performance while zeroing out additional inference overhead. Extensive experiments across $4$ state-of-the-art (SOTA) video DMs, with parameter sizes ranging from $1.3\text{B}\sim14\text{B}$, show that QVGen is the first to reach full-precision comparable quality under 4-bit settings. Moreover, it significantly outperforms existing methods. For instance, our 3-bit CogVideoX-2B achieves improvements of $+25.28$ in Dynamic Degree and $+8.43$ in Scene Consistency on VBench. Code and models are available at https://github.com/ModelTC/QVGen.

QVGen: Pushing the Limit of Quantized Video Generative Models

TL;DR

QVGen addresses the challenge of deploying ultra‑low‑bit video diffusion models by introducing a quantization‑aware training framework that uses learnable auxiliary modules Φ to stabilize convergence. A theoretical regret analysis shows that minimizing the gradient norm ||g_t||_2 is key for QAT stability, which Φ helps achieve. To avoid inference overhead, QVGen employs a rank‑decay schedule that progressively removes Φ via SVD and a rank‑based regularization, yielding a standard 4‑bit quantized model with minimal performance loss. Extensive experiments across CogVideoX and Wan families demonstrate state‑of‑the‑art 3‑bit performance and full‑precision parity at 4‑bit, with substantial memory savings and practical efficiency gains for real‑world deployment.

Abstract

Video diffusion models (DMs) have enabled high-quality video synthesis. Yet, their substantial computational and memory demands pose serious challenges to real-world deployment, even on high-end GPUs. As a commonly adopted solution, quantization has proven notable success in reducing cost for image DMs, while its direct application to video DMs remains ineffective. In this paper, we present QVGen, a novel quantization-aware training (QAT) framework tailored for high-performance and inference-efficient video DMs under extremely low-bit quantization (e.g., 4-bit or below). We begin with a theoretical analysis demonstrating that reducing the gradient norm is essential to facilitate convergence for QAT. To this end, we introduce auxiliary modules () to mitigate large quantization errors, leading to significantly enhanced convergence. To eliminate the inference overhead of , we propose a rank-decay strategy that progressively eliminates . Specifically, we repeatedly employ singular value decomposition (SVD) and a proposed rank-based regularization to identify and decay low-contributing components. This strategy retains performance while zeroing out additional inference overhead. Extensive experiments across state-of-the-art (SOTA) video DMs, with parameter sizes ranging from , show that QVGen is the first to reach full-precision comparable quality under 4-bit settings. Moreover, it significantly outperforms existing methods. For instance, our 3-bit CogVideoX-2B achieves improvements of in Dynamic Degree and in Scene Consistency on VBench. Code and models are available at https://github.com/ModelTC/QVGen.
Paper Structure (38 sections, 3 theorems, 35 equations, 16 figures, 28 tables, 1 algorithm)

This paper contains 38 sections, 3 theorems, 35 equations, 16 figures, 28 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. QVGen
  4. Improving Convergence with Auxiliary Modules Phi
  5. Progressively Shrinking Phi via Rank-Decay
  6. Experiments
  7. Implementation Details
  8. Performance Analysis
  9. Ablation Studies
  10. Efficiency Discussion
  11. Related Work
  12. Conclusions and Limitations
  13. Alogrithm of QVGen
  14. Proof of Thm. \ref{['theorem:regret']}
  15. Convergence Analysis without Requiring Convexity
  16. ...and 23 more sections

Key Result

Theorem 3.1

Assume that $f_t$ is convexThis may not hold for deep networks. Therefore, we also provide a nonconvex convergence analysis in Sec. sec:nonconvex. and $\forall {\bm{\theta}}_i, {\bm{\theta}}_j\in\mathbb{S}^d, \|{\bm{\theta}}_i-{\bm{\theta}}_j\|_{\infty}\leq D_{\infty}$. Then the average regret is up

Figures (16)

  • Figure 1: Comparison of samples generated by CogVideoX-$2$B yang2025cogvideoxtexttovideodiffusionmodels with a fixed random seed. "W$x$A$y$" denotes "$x$"-bit per-channel weight and "$y$"-bit per-token activation quantization. Our approach far outperforms previous PTQ (i.e., (f)) and QAT (i.e., (c)-(e)) methods. To be noted, methods (c)-(f) have achieved noticeable performance for $4$-bit image DMs. More visual results can be found in Sec. \ref{['sec:vis']}.
  • Figure 2: Overview of the proposed QVGen. (a) This framework integrates auxiliary modules $\Phi$ to improve training convergence (Sec. \ref{['sec:module']}). (b) To maintain performance while eliminating inference overhead induced by $\Phi$, we design a rank-decay schedule that progressively shrinks the entire $\Phi$ to $\varnothing$ through iteratively applying the following two strategies (Sec. \ref{['sec:shrink']}): (i) SVD to identify the low-impact components in $\Phi$; (ii) A rank-based regularization ${\bm{\gamma}}$ to decay the identified components to $\varnothing$. A detailed procedure can be found in Sec. \ref{['sec:algo']}.
  • Figure 3: (Upper) $\|{\bm{g}}_t\|_2$vs. #steps and (Lower) training loss (i.e., Eq. (\ref{['eq:loss']})) vs. #steps across different video DMs and $4$-bit QAT methods."$\Phi$" denotes our approach in Sec. \ref{['sec:module']}.
  • Figure 4: Singular value variation in ${\mathbf{W}}_{\Phi}$ across training iterations for $4$-bit video DMs. We visualize the average of the singular values $\{\sigma_s\}_{s=1,2,\ldots, 2^{10}} \cup \{\sigma_d\}$ across layers of all Attention blocks vaswani2023attentionneed and feed-forward networks (FFNs), respectively. "$0$ step" denotes the initialization state before QAT.
  • Figure 5: Performance for huge video DMs on VBench-2.0 zheng2025vbench20advancingvideogeneration. Our $4$-bit models exhibit a minimal drop of ${\sim}1\%$ in total score.
  • ...and 11 more figures

Theorems & Definitions (6)

  • Theorem 3.1
  • Theorem B.3
  • proof
  • Theorem C.4: Convergence to a first-order stationary point
  • proof
  • Remark C.5: Connection to our convex analysis