Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement

ZhanFeng Feng, Long Peng, Xin Di, Yong Guo, Wenbo Li, Yulun Zhang, Renjing Pei, Yang Wang, Yang Cao, Zheng-Jun Zha

TL;DR

PMQ-VE tackles the challenge of deploying Transformer-based multi-frame video enhancement models under low-bit quantization. It introduces a coarse-to-fine framework with Backtracking-based Multi-Frame Quantization (BMFQ) to assign frame-specific clipping bounds and Progressive Multi-Teacher Distillation (PMTD) to bridge the capacity gap between low-bit students and multi-level teachers. By evaluating on STVSR, VSR, and VFI benchmarks, PMQ-VE achieves state-of-the-art PSNR/SSIM and perceptual metrics across multiple bit-widths, while preserving fine details and temporal coherence. The approach enables practical, edge-level deployment of high-quality multi-frame video enhancement systems with tangible efficiency gains and robust performance.

Abstract

Multi-frame video enhancement tasks aim to improve the spatial and temporal resolution and quality of video sequences by leveraging temporal information from multiple frames, which are widely used in streaming video processing, surveillance, and generation. Although numerous Transformer-based enhancement methods have achieved impressive performance, their computational and memory demands hinder deployment on edge devices. Quantization offers a practical solution by reducing the bit-width of weights and activations to improve efficiency. However, directly applying existing quantization methods to video enhancement tasks often leads to significant performance degradation and loss of fine details. This stems from two limitations: (a) inability to allocate varying representational capacity across frames, which results in suboptimal dynamic range adaptation; (b) over-reliance on full-precision teachers, which limits the learning of low-bit student models. To tackle these challenges, we propose a novel quantization method for video enhancement: Progressive Multi-Frame Quantization for Video Enhancement (PMQ-VE). This framework features a coarse-to-fine two-stage process: Backtracking-based Multi-Frame Quantization (BMFQ) and Progressive Multi-Teacher Distillation (PMTD). BMFQ utilizes a percentile-based initialization and iterative search with pruning and backtracking for robust clipping bounds. PMTD employs a progressive distillation strategy with both full-precision and multiple high-bit (INT) teachers to enhance low-bit models' capacity and quality. Extensive experiments demonstrate that our method outperforms existing approaches, achieving state-of-the-art performance across multiple tasks and benchmarks.The code will be made publicly available at: https://github.com/xiaoBIGfeng/PMQ-VE.

PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement

TL;DR

PMQ-VE tackles the challenge of deploying Transformer-based multi-frame video enhancement models under low-bit quantization. It introduces a coarse-to-fine framework with Backtracking-based Multi-Frame Quantization (BMFQ) to assign frame-specific clipping bounds and Progressive Multi-Teacher Distillation (PMTD) to bridge the capacity gap between low-bit students and multi-level teachers. By evaluating on STVSR, VSR, and VFI benchmarks, PMQ-VE achieves state-of-the-art PSNR/SSIM and perceptual metrics across multiple bit-widths, while preserving fine details and temporal coherence. The approach enables practical, edge-level deployment of high-quality multi-frame video enhancement systems with tangible efficiency gains and robust performance.

Abstract

Multi-frame video enhancement tasks aim to improve the spatial and temporal resolution and quality of video sequences by leveraging temporal information from multiple frames, which are widely used in streaming video processing, surveillance, and generation. Although numerous Transformer-based enhancement methods have achieved impressive performance, their computational and memory demands hinder deployment on edge devices. Quantization offers a practical solution by reducing the bit-width of weights and activations to improve efficiency. However, directly applying existing quantization methods to video enhancement tasks often leads to significant performance degradation and loss of fine details. This stems from two limitations: (a) inability to allocate varying representational capacity across frames, which results in suboptimal dynamic range adaptation; (b) over-reliance on full-precision teachers, which limits the learning of low-bit student models. To tackle these challenges, we propose a novel quantization method for video enhancement: Progressive Multi-Frame Quantization for Video Enhancement (PMQ-VE). This framework features a coarse-to-fine two-stage process: Backtracking-based Multi-Frame Quantization (BMFQ) and Progressive Multi-Teacher Distillation (PMTD). BMFQ utilizes a percentile-based initialization and iterative search with pruning and backtracking for robust clipping bounds. PMTD employs a progressive distillation strategy with both full-precision and multiple high-bit (INT) teachers to enhance low-bit models' capacity and quality. Extensive experiments demonstrate that our method outperforms existing approaches, achieving state-of-the-art performance across multiple tasks and benchmarks.The code will be made publicly available at: https://github.com/xiaoBIGfeng/PMQ-VE.
Paper Structure (16 sections, 5 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 16 sections, 5 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Video Enhancement
  4. Model Quantization
  5. Methodology
  6. Problem Formulation
  7. Observations and Motivation
  8. Proposed Method
  9. Backtracking-based Multi-Frame Quantization
  10. Progressive Multi-Teacher Distillation
  11. Experiment and Analysis
  12. Experiment Setting
  13. Quantitative Results
  14. Qualitative Results
  15. Ablation Studies
  16. ...and 1 more sections

Figures (4)

  • Figure 1: (a) Qualitative comparison of reconstructed frames using different quantization methods. Quantitative comparison of PSNR(b) and SSIM(c) improvements across three video enhancement tasks (STVSR, VFI, VSR). Our method consistently outperforms existing quantization approaches in both visual quality and quantitative metrics.
  • Figure 2: Finding and Motivation. (a) In multi-frame video enhancement, activation distributions vary significantly across frames. Traditional per-tensor quantization (b) fails to dynamically adjust quantization bounds for these variations, but our method (c) achieves this dynamic adjustment. (d) We calculated PSNR and residual maps for FP, INT8, INT4, and INT2 with respect to GT. The significant gap between low-bit (INT2/4) and full-precision (FP) model suggests that low-bit struggles to learn directly from FP. This inspired us to use multiple teacher models for supervision.
  • Figure 3: The overall framework of our proposed method.
  • Figure 4: Visual comparisons under 4-bit quantization for three video enhancement tasks: from top to bottom are STVSR, VSR, and VFI tasks. More results are provided in the Appendix.