Temporal Inconsistency Guidance for Super-resolution Video Quality Assessment

TL;DR

The paper tackles SR video quality assessment by identifying temporal inconsistency as a key perceptual cue amplified by SR. It introduces TIG-SVQA, which leverages a temporal inconsistency map to guide both spatial (IHSM with a DW-SA Transformer and fine ResNet) and temporal (IGTM with a Visual Memory Capacity Block and two-stage aggregation) modeling, producing scores via S1 and S2 fusion. Empirical results on SFD, MFD, and Combined-VSR datasets show strong improvements over SR/IQA/VQA baselines, and the model demonstrates robust generalization to UGC datasets and fused test sets. The work also analyzes loss functions, memory-adaptive mechanisms, and cross-dataset transfer, highlighting practical implications for perceptual SR video quality prediction and providing code for reproducibility.

Abstract

As super-resolution (SR) techniques introduce unique distortions that fundamentally differ from those caused by traditional degradation processes (e.g., compression), there is an increasing demand for specialized video quality assessment (VQA) methods tailored to SR-generated content. One critical factor affecting perceived quality is temporal inconsistency, which refers to irregularities between consecutive frames. However, existing VQA approaches rarely quantify this phenomenon or explicitly investigate its relationship with human perception. Moreover, SR videos exhibit amplified inconsistency levels as a result of enhancement processes. In this paper, we propose \textit{Temporal Inconsistency Guidance for Super-resolution Video Quality Assessment (TIG-SVQA)} that underscores the critical role of temporal inconsistency in guiding the quality assessment of SR videos. We first design a perception-oriented approach to quantify frame-wise temporal inconsistency. Based on this, we introduce the Inconsistency Highlighted Spatial Module, which localizes inconsistent regions at both coarse and fine scales. Inspired by the human visual system, we further develop an Inconsistency Guided Temporal Module that performs progressive temporal feature aggregation: (1) a consistency-aware fusion stage in which a visual memory capacity block adaptively determines the information load of each temporal segment based on inconsistency levels, and (2) an informative filtering stage for emphasizing quality-related features. Extensive experiments on both single-frame and multi-frame SR video scenarios demonstrate that our method significantly outperforms state-of-the-art VQA approaches. The code is publicly available at https://github.com/Lighting-YXLI/TIG-SVQA-main.

Figures (10)

• Figure 1: Visualizations for comparing temporal inconsistency with motion. Rows (a) and (c) show consecutive frames of a reference video and SR video, respectively. (b) and (d) are the optical flow of (a) and (c), respectively. (e) is the temporal inconsistency information for (c).
• Figure 2: Correlation and performance comparison between motion and motion difference for model guidance. We analyze the correlation (SRCC) between perceptual quality (1–MOS) and motion or motion difference based on video complexity. Performance comparison shows that motion achieves SRCC/PLCC of 0.885/0.913, while motion difference reaches 0.939/0.942 on the Combined-VSR dataset.
• Figure 3: The framework of the proposed TIG-SVQA. The spatial module computes temporal inconsistency and applies pixel-level weighting at both coarse and fine granularities to emphasize inconsistent regions. The temporal module involves two stages: Consistency-aware Fusion and Informative Filtering.
• Figure 4: The details of the proposed Deformable Window Super-Attention (DW-SA) Transformer block, which adaptively adjusts window locations, up-samples features within each window, and then shifts the windows.
• Figure 5: The framework of Inconsistency Highlighted Spatial Module (IHSM) includes detailed structures for both the Coarse Scene Spatial Extractor and the Fine Scene Spatial Extractor. Each layer of these extractors, along with the input and output feature dimensions, is described. The input frame batch for the $j$-th iteration is denoted as $F_{B}^{j}$, where $B$ represents the batch size, and $W$ and $H$ correspond to the frame's width and height, respectively.