LSGQuant: Layer-Sensitivity Guided Quantization for One-Step Diffusion Real-World Video Super-Resolution

TL;DR

LSGQuant tackles the heavy cost of one-step diffusion video super-resolution by enabling effective low-bit quantization through three components: Dynamic Range Adaptive Quantizer (DRAQ) to align activation ranges, Variance-Oriented Layer Training Strategy (VOLTS) to allocate training focus by layer sensitivity, and Quantization-Aware Alternating Optimization (QAO) to jointly optimize a low-bit quantized branch with a high-precision residual path. The method adopts a two-branch DiT quantization layer and applies it to a WAN2.1-based one-step diffusion backbone, achieving near full-precision performance at 4-bit while significantly reducing Params and Ops. Extensive ablations confirm that DRAQ, VOLTS, and QAO each contribute to robustness and perceptual quality, with 4-bit LSGQuant surpassing prior PTQ methods across synthetic and real-world VSR benchmarks. The work enables practical deployment of diffusion-based VSR on resource-constrained devices with real-world degradations, narrowing the gap between high-fidelity results and efficient inference.

Abstract

One-Step Diffusion Models have demonstrated promising capability and fast inference in video super-resolution (VSR) for real-world. Nevertheless, the substantial model size and high computational cost of Diffusion Transformers (DiTs) limit downstream applications. While low-bit quantization is a common approach for model compression, the effectiveness of quantized models is challenged by the high dynamic range of input latent and diverse layer behaviors. To deal with these challenges, we introduce LSGQuant, a layer-sensitivity guided quantizing approach for one-step diffusion-based real-world VSR. Our method incorporates a Dynamic Range Adaptive Quantizer (DRAQ) to fit video token activations. Furthermore, we estimate layer sensitivity and implement a Variance-Oriented Layer Training Strategy (VOLTS) by analyzing layer-wise statistics in calibration. We also introduce Quantization-Aware Optimization (QAO) to jointly refine the quantized branch and a retained high-precision branch. Extensive experiments demonstrate that our method has nearly performance to origin model with full-precision and significantly exceeds existing quantization techniques. Code is available at: https://github.com/zhengchen1999/LSGQuant.

Figures (6)

• Figure 1: Low-bit LSGQuant comparison with bfloat16 full-precision model, the recent leading quantization methods ViDiT-Q zhao2024vidit and SVDQuant li2024svdquant.
• Figure 2: Performance comparisons on the synthetic REDS30 dataset nah2019ntire between different quantization methods in 4-bit and 6-bit scenarios. LSGQuant achieves leading scores on all evaluation metrics.
• Figure 3: Overview of our LSGQuant. Firstly, we collect per-channel layer statistics by a single inference process. Next, we calculate layer sensitivity and estimate its importance to the final video output. Finally, we allocate training iterations by estimated sensitivities.
• Figure 4: Input data sampled from cross-attention query layer. (a) Statistical distribution of activation values. While majority of values are concentrated near zero, a significant long-tail distribution exists. These extreme values would dominate the quantization range. (b) Distribution of input latent values across different blocks. The statistical characteristics of outliers varies with depth.
• Figure 5: We conduct 4-bit scenarios visual exhibition for synthetic (REDS30 nah2019ntire) and real-world (MVSR4x wang2023benchmark) datasets. In 4-bit settings, our approach significantly surpasses recent existing methods.