DiffVQA: Video Quality Assessment Using Diffusion Feature Extractor

TL;DR

This paper tackles the mismatch between no-reference VQA methods and human perceptual quality under diverse real-world distortions by leveraging diffusion priors. By adapting a pre-trained diffusion model with a ControlNet-like Controller to reconstruct identical frames, and extracting semantic and distortion features from a resizing and a cropping branch respectively, DiffVQA achieves rich frame representations. A parallel Mamba-based Temporal Coherence Augmentation Block enhances long-range temporal dynamics, and feature fusion yields a final quality score that generalizes well across datasets, outperforming CNN and ViT backbones in intra- and cross-dataset evaluations. The approach demonstrates that diffusion-based representations can improve VQA accuracy and generalization, offering a practical route to more perceptually aligned video quality assessment in real-world conditions.

Abstract

Video Quality Assessment (VQA) aims to evaluate video quality based on perceptual distortions and human preferences. Despite the promising performance of existing methods using Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), they often struggle to align closely with human perceptions, particularly in diverse real-world scenarios. This challenge is exacerbated by the limited scale and diversity of available datasets. To address this limitation, we introduce a novel VQA framework, DiffVQA, which harnesses the robust generalization capabilities of diffusion models pre-trained on extensive datasets. Our framework adapts these models to reconstruct identical input frames through a control module. The adapted diffusion model is then used to extract semantic and distortion features from a resizing branch and a cropping branch, respectively. To enhance the model's ability to handle long-term temporal dynamics, a parallel Mamba module is introduced, which extracts temporal coherence augmented features that are merged with the diffusion features to predict the final score. Experiments across multiple datasets demonstrate DiffVQA's superior performance on intra-dataset evaluations and its exceptional generalization across datasets. These results confirm that leveraging a diffusion model as a feature extractor can offer enhanced VQA performance compared to CNN and ViT backbones.

Figures (7)

• Figure 1: PLCC and SRCC performance across intra- and cross-dataset evaluations. DiffVQA, represented by red triangles, shows superior performance against established methods like DOVER wu2023exploring, FAST-VQA wu2022fast, TLVQM korhonen2019two, VIDEVAL tu2021ugc, and Patch-VQ ying2021patch. We evaluate DiffVQA on the KoNViD-1k hosu2017konstanz dataset using models trained on KoNViD-1k for intra-dataset evaluation and LSVQ ying2021patch for cross-dataset evaluation, demonstrating its robust generalization capabilities across different datasets.
• Figure 2: Illustration of Bidirectional Mamba Block. The initial normalization and the final residual are omitted for simplification.
• Figure 3: Adaptation process of the Diffusion Feature Extractor shown in (a). $\mathcal{L}_{\text{Adap}}$ is computed between the added noise $\epsilon$ and the predicted noise $\epsilon_{\theta}$. (b) A similar architecture can be used for image restoration, but we repurpose it for image reconstruction here. (c) During inference, we use $\hat{z}_0$, along with the features from Denoising Network at time step $t = 0$ as the extracted features.
• Figure 4: Examples of reconstructed results from the adapted diffusion model, along with error maps showing the differences between the input and reconstructed frames. After adaptation, the diffusion model reconstructs the input frames with minimal error.
• Figure 5: Architecture of DiffVQA. The diffusion feature extractor extracts semantic and distortion features from video frames, which are enhanced by the DFF, TDM, and FFF modules. The TCAB are also used to capture temporal coherence in parallel. Features extracted from the resized branch are denoted with the suffix ‘‘-S’’ to represent semantic4 information, while those from the random crop branch use the suffix ‘‘-D’’ for distortion-related features.