Convolutions Need Registers Too: HVS-Inspired Dynamic Attention for Video Quality Assessment

TL;DR

This work tackles no-reference video quality assessment by introducing DAGR-VQA, a framework that embeds register-tokens directly into a convolutional backbone to instantiate dynamic, HVS-inspired attention without relying on heavy self-attention or motion estimation. The solution comprises a saliency pretraining stage using a UNet3D augmented with register-tokens and a spatio-temporal VQA module that fuses saliency-guided spatial features with a lightweight temporal transformer to predict perceptual quality. Key contributions include (i) a register-token–augmented saliency pretraining pipeline, (ii) a modular VQA architecture that combines spatial saliency, temporal modeling, and robust MOS-based training, and (iii) extensive experiments across four NR-VQA benchmarks demonstrating competitive accuracy and real-time performance (about 387.7 FPS at 1080p). Ablation studies show that register-tokens promote stable, temporally coherent attention, and act synergistically with dynamic saliency to improve VQA predictions. The approach offers a practical path toward real-time video quality monitoring in streaming systems, with potential for adaptive saliency fusion and broader applications in perceptual video analysis.

Abstract

No-reference video quality assessment (NR-VQA) estimates perceptual quality without a reference video, which is often challenging. While recent techniques leverage saliency or transformer attention, they merely address global context of the video signal by using static maps as auxiliary inputs rather than embedding context fundamentally within feature extraction of the video sequence. We present Dynamic Attention with Global Registers for Video Quality Assessment (DAGR-VQA), the first framework integrating register-token directly into a convolutional backbone for spatio-temporal, dynamic saliency prediction. By embedding learnable register tokens as global context carriers, our model enables dynamic, HVS-inspired attention, producing temporally adaptive saliency maps that track salient regions over time without explicit motion estimation. Our model integrates dynamic saliency maps with RGB inputs, capturing spatial data and analyzing it through a temporal transformer to deliver a perceptually consistent video quality assessment. Comprehensive tests conducted on the LSVQ, KonVid-1k, LIVE-VQC, and YouTube-UGC datasets show that the performance is highly competitive, surpassing the majority of top baselines. Research on ablation studies demonstrates that the integration of register tokens promotes the development of stable and temporally consistent attention mechanisms. Achieving an efficiency of 387.7 FPS at 1080p, DAGR-VQA demonstrates computational performance suitable for real-time applications like multimedia streaming systems.

Figures (8)

• Figure 1: Overview of DAGR-VQA. During pretraining, RGB frames from saliency dataset ($B_1 \times 3 \times T_1 \times H_1 \times W_1$) are concatenated ($\bigoplus$) with register-tokens and processed by a UNet3D. The pretrained model inferences saliency maps ($B_2 \times 1 \times T_2 \times H_2 \times W_2$) for VQA videos. For each frame, fusion is performed as $(1-\alpha)$ times original frames, added ($\bigoplus$) with $\alpha$ times saliency map element-wise multiplied ($\bigotimes$) original frames. Spatial, temporal, and regression modules complete the VQA pipeline.
• Figure 2: Saliency pretraining architecture with register-tokens. Sampled input frames are concatenated ($\oplus$) with learnable register-tokens and processed by a UNet3D-style encoder-decoder. The network applies successive 3D convolutions, pooling, a bottleneck, attention, and adaptive pooling to produce temporally stable, quality-aware saliency maps. Channel and spatial dimensions at each stage are indicated above the blocks.
• Figure 3: Overview of the proposed spatio-temporal VQA module. A pretrained saliency model $f_{\theta}()$ produces dynamic saliency maps to form saliency-weighted video frames, which pass through a ResNet-50 backbone to extract and pool spatial features $F_{t}$. These are temporally pooled to $y_{s}$. Then, processed by a Transformer encoder, the sequence is concatenated and projected with pooled temporal features $y_{t}$ before being fed to a regressor $f_{\omega}$, which predicts the video quality score $\hat{y}$. LN denotes layer normalization, FFN is Feed Forward Network, MHSA is multi-head self-attention, Conv is the initial convolution, Block 1–4 means the ResNet convolution stages, and FC means a fully connected layer.
• Figure 4: 5-Fold Cross-Database SRCC (Train: LSVQ $\rightarrow$ Test: KoNViD-1k / LIVEVQC). Each box plot shows results for a VQA method, where the central black line within each box indicates the median SRCC across 5 runs.
• Figure 5: Ablation study: SRCC of configurations with/without dynamic/static saliency and register-tokens across four datasets. Higher is better.