QVD: Post-training Quantization for Video Diffusion Models

TL;DR

The paper addresses the large memory and latency demands of video diffusion models by introducing QVD, a post-training quantization framework tailored for video data. It jointly tackles temporal feature skew with High Temporal Discriminability Quantization (HTDQ) and inter-channel activation disparities with Scattered Channel Range Integration (SCRI). Key contributions include the Temporal Discriminability Score (TDScore) and the HiDi-TQ quantizer for temporal features, along with a per-channel activation range integration strategy that enhances quantization level coverage. Empirical results demonstrate near-lossless performance at 8-bit precision on multiple datasets and a substantial improvement in Fréchet Video Distance (FVD) over image-domain PTQ baselines, enabling practical deployment of video diffusion models.

Abstract

Recently, video diffusion models (VDMs) have garnered significant attention due to their notable advancements in generating coherent and realistic video content. However, processing multiple frame features concurrently, coupled with the considerable model size, results in high latency and extensive memory consumption, hindering their broader application. Post-training quantization (PTQ) is an effective technique to reduce memory footprint and improve computational efficiency. Unlike image diffusion, we observe that the temporal features, which are integrated into all frame features, exhibit pronounced skewness. Furthermore, we investigate significant inter-channel disparities and asymmetries in the activation of video diffusion models, resulting in low coverage of quantization levels by individual channels and increasing the challenge of quantization. To address these issues, we introduce the first PTQ strategy tailored for video diffusion models, dubbed QVD. Specifically, we propose the High Temporal Discriminability Quantization (HTDQ) method, designed for temporal features, which retains the high discriminability of quantized features, providing precise temporal guidance for all video frames. In addition, we present the Scattered Channel Range Integration (SCRI) method which aims to improve the coverage of quantization levels across individual channels. Experimental validations across various models, datasets, and bit-width settings demonstrate the effectiveness of our QVD in terms of diverse metrics. In particular, we achieve near-lossless performance degradation on W8A8, outperforming the current methods by 205.12 in FVD.

Figures (7)

• Figure 1: Comparison of image diffusion (left) and video diffusion (right). (a) Features of all frames rely on the same temporal feature in VDMs. (b) Significantly inter-channel variation issue occurs in temporal attention modules.
• Figure 2: Overview of QVD. The left is the High Temporal Discriminability Quantization, which uses the HiDi-TQ quantizer to retain the low TDScore of temporal features. The red arrow points to the location of the outlier. The right is the Scattered Channel Range Integration, which aims at mitigating the discreteness and asymmetry in inter-channel activation ranges, thereby enhancing the utilization rate of quantization levels by individual channels.
• Figure 3: Heat maps of full-precision temporal feature and its quantized versions of the uniform quantizer and the HTDQ.
• Figure 4: Histogram of the temporal feature. (a) shows the histogram of 100% data of a temporal feature. (b) presents the middle 90% of the data which is concentrated within the range of $\left [-0.002, 0.002 \right ]$ and a single blue rectangle in the background indicates that these data are mapped to the same value. (c) depicts the distribution of the middle 90% of data processed through a logarithmic function, covering 10 quantization levels.
• Figure 5: Performance of various quantization strategies. The left axis represents the TDScore. The right axis delineates the L2 quantization loss.