Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Video Quality Assessment from the Perspective of Generalization

Xinli Yue, Jianhui Sun, Liangchao Yao, Fan Xia, Yuetang Deng, Tianyi Wang, Lei Li, Fengyun Rao, Jing Lv, Qian Wang, Lingchen Zhao

TL;DR

The results reveal that adversarial weight perturbations can effectively smooth this landscape of VQA models, significantly improving the generalization performance, with cross-dataset generalization and fine-tuning performance enhanced by up to 1.8% and 3%, respectively.

Abstract

The increasing popularity of short video platforms such as YouTube Shorts, TikTok, and Kwai has led to a surge in User-Generated Content (UGC), which presents significant challenges for the generalization performance of Video Quality Assessment (VQA) tasks. These challenges not only affect performance on test sets but also impact the ability to generalize across different datasets. While prior research has primarily focused on enhancing feature extractors, sampling methods, and network branches, it has largely overlooked the generalization capabilities of VQA tasks. In this work, we reevaluate the VQA task from a generalization standpoint. We begin by analyzing the weight loss landscape of VQA models, identifying a strong correlation between this landscape and the generalization gaps. We then investigate various techniques to regularize the weight loss landscape. Our results reveal that adversarial weight perturbations can effectively smooth this landscape, significantly improving the generalization performance, with cross-dataset generalization and fine-tuning performance enhanced by up to 1.8% and 3%, respectively. Through extensive experiments across various VQA methods and datasets, we validate the effectiveness of our approach. Furthermore, by leveraging our insights, we achieve state-of-the-art performance in Image Quality Assessment (IQA) tasks. Our code is available at https://github.com/XinliYue/VQA-Generalization.

Revisiting Video Quality Assessment from the Perspective of Generalization

TL;DR

The results reveal that adversarial weight perturbations can effectively smooth this landscape of VQA models, significantly improving the generalization performance, with cross-dataset generalization and fine-tuning performance enhanced by up to 1.8% and 3%, respectively.

Abstract

The increasing popularity of short video platforms such as YouTube Shorts, TikTok, and Kwai has led to a surge in User-Generated Content (UGC), which presents significant challenges for the generalization performance of Video Quality Assessment (VQA) tasks. These challenges not only affect performance on test sets but also impact the ability to generalize across different datasets. While prior research has primarily focused on enhancing feature extractors, sampling methods, and network branches, it has largely overlooked the generalization capabilities of VQA tasks. In this work, we reevaluate the VQA task from a generalization standpoint. We begin by analyzing the weight loss landscape of VQA models, identifying a strong correlation between this landscape and the generalization gaps. We then investigate various techniques to regularize the weight loss landscape. Our results reveal that adversarial weight perturbations can effectively smooth this landscape, significantly improving the generalization performance, with cross-dataset generalization and fine-tuning performance enhanced by up to 1.8% and 3%, respectively. Through extensive experiments across various VQA methods and datasets, we validate the effectiveness of our approach. Furthermore, by leveraging our insights, we achieve state-of-the-art performance in Image Quality Assessment (IQA) tasks. Our code is available at https://github.com/XinliYue/VQA-Generalization.
Paper Structure (38 sections, 14 equations, 4 figures, 13 tables, 3 algorithms)

This paper contains 38 sections, 14 equations, 4 figures, 13 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Quality Assessment (VQA)
  4. Generalization
  5. Weight Perturbation
  6. Relationship Between Weight Loss Landscape and VQA Generalization Gap
  7. Weight Loss Landscape Visualization
  8. Generalization Gap During Training
  9. Generalization Gap Across Different Methods
  10. Reducing the Generalization Gap
  11. $L_2$ Regularization
  12. Data Augmentation
  13. Weight Perturbation
  14. Perturbation Direction
  15. Perturbation Magnitude
  16. ...and 23 more sections

Figures (4)

  • Figure 1: The training results of various methods on the KoNViD-1k dataset konvid. (a) and (b) illustrate the performance metrics of FAST-VQA fastvqa on both the training and testing sets across different epochs, as well as the generalization gap. (c) compares the weight loss landscape of FAST-VQA at different epochs. (d) contrasts the weight loss landscapes of different VQA methods.
  • Figure 2: Weight loss landscape of FAST-VQA on the KoNViD-1k dataset under various modifications. (a) Different $L_2$ regularization parameters $\lambda$, (b) Different data augmentation methods, (c) Different RWP perturbation magnitudes, (d) Different AWP perturbation magnitudes.
  • Figure 3: The training results of various methods on the YouTubeUGC dataset. (a) illustrates the performance metrics of FAST-VQA on both the training and testing sets across different epochs, as well as the generalization gap. (b) compares the weight loss landscape of FAST-VQA at different epochs.
  • Figure 4: Weight loss landscape of FAST-VQA on the YouTubeUGC dataset under various modifications. (a) Different $L_2$ regularization parameters $\lambda$, (b) Different data augmentation methods, (c) Different RWP perturbation magnitudes, (d) Different AWP perturbation magnitudes.