KVQ: Boosting Video Quality Assessment via Saliency-guided Local Perception

TL;DR

This paper tackles NR VQA by recognizing that video quality varies across regions due to distortions and saliency. It introduces KVQ, a framework built on a Video Swin Transformer backbone with a dual-branch design that predicts saliency and local texture, guided by Fusion-Window Attention (FWA) and a Local Perception Constraint (LPC). KVQ formalizes the global quality as a saliency-weighted measure of local texture and enforces region-wise texture independence to improve local perception; it also ensembles multi-scale saliency maps to robustly capture attention. Empirical results on LSVQ, KoNViD-1k, LIVE-VQC, and YouTube-UGC show state-of-the-art performance in intra-dataset, cross-dataset, and transfer settings, while the LPVQ dataset provides region-wise local quality annotations for validating local perception. The approach offers practical benefits for fine-grained quality assessment and potential guidance for region-aware encoding and enhancement strategies.

Abstract

Video Quality Assessment (VQA), which intends to predict the perceptual quality of videos, has attracted increasing attention. Due to factors like motion blur or specific distortions, the quality of different regions in a video varies. Recognizing the region-wise local quality within a video is beneficial for assessing global quality and can guide us in adopting fine-grained enhancement or transcoding strategies. Due to the heavy cost of annotating region-wise quality, the lack of ground truth constraints from relevant datasets further complicates the utilization of local perception. Inspired by the Human Visual System (HVS) that links global quality to the local texture of different regions and their visual saliency, we propose a Kaleidoscope Video Quality Assessment (KVQ) framework, which aims to effectively assess both saliency and local texture, thereby facilitating the assessment of global quality. Our framework extracts visual saliency and allocates attention using Fusion-Window Attention (FWA) while incorporating a Local Perception Constraint (LPC) to mitigate the reliance of regional texture perception on neighboring areas. KVQ obtains significant improvements across multiple scenarios on five VQA benchmarks compared to SOTA methods. Furthermore, to assess local perception, we establish a new Local Perception Visual Quality (LPVQ) dataset with region-wise annotations. Experimental results demonstrate the capability of KVQ in perceiving local distortions. KVQ models and the LPVQ dataset will be available at https://github.com/qyp2000/KVQ.

Figures (7)

• Figure 1: Human visual saliency and different attention mechanisms. (Left) Visual saliency obtained through mouse-tracking in the SALICON jiang2015salicon dataset. (Middle) Window Attention mechanism DBLP:conf/iccv/LiuL00W0LG21, where patches establish connections only within the window. (Right) Our Fusion-Window Attention, where patches can allocate long-range attention across correlated regions.
• Figure 2: Perceptual quality of different regions varies due to factors (e.g., texture, motion, compression). These variations may not necessarily align with the global quality.
• Figure 3: The framework of our proposed KVQ.
• Figure 4: Illustration of Local Perception Constraint.
• Figure 5: Examples and the MOS distribution in the proposed LPVQ dataset. Please zoom in for a better view.