No-Reference Rendered Video Quality Assessment: Dataset and Metrics

TL;DR

The paper tackles NR-VQA for rendered videos by introducing ReVQ-2k, a large dataset with OA-MOS and TS-MOS annotations, and a two-stream NR-VQA metric that jointly assesses image quality and temporal stability. The image-quality stream reuses FAST-VQA with a Swin-T backbone, while the temporal-stability stream uses motion estimation and multi-time-scale differencing, fused by an MLP to predict $q^{pred}$. Experimental results show the proposed method outperforms existing NR-VQA baselines on rendered content and remains robust on camera-captured datasets, with TS-MOS supervision delivering additional accuracy gains. The work also demonstrates practical utility in benchmarking mobile supersampling and evaluating frame-generation strategies, providing a valuable benchmark and baseline for NR-VQA of rendered videos.

Abstract

Quality assessment of videos is crucial for many computer graphics applications, including video games, virtual reality, and augmented reality, where visual performance has a significant impact on user experience. When test videos cannot be perfectly aligned with references or when references are unavailable, the significance of no-reference video quality assessment (NR-VQA) methods is undeniable. However, existing NR-VQA datasets and metrics are primarily focused on camera-captured videos; applying them directly to rendered videos would result in biased predictions, as rendered videos are more prone to temporal artifacts. To address this, we present a large rendering-oriented video dataset with subjective quality annotations, as well as a designed NR-VQA metric specific to rendered videos. The proposed dataset includes a wide range of 3D scenes and rendering settings, with quality scores annotated for various display types to better reflect real-world application scenarios. Building on this dataset, we calibrate our NR-VQA metric to assess rendered video quality by looking at both image quality and temporal stability. We compare our metric to existing NR-VQA metrics, demonstrating its superior performance on rendered videos. Finally, we demonstrate that our metric can be used to benchmark supersampling methods and assess frame generation strategies in real-time rendering.

Figures (8)

• Figure 1: Process for assessing the quality of rendered videos. Rendered videos undergo a complex transition from 3D scenes to on-screen displays. During this transition, 3D scene quality, rendering configurations, and display types all have a significant impact on perceived quality.
• Figure 2: Examples of 3D scenes from our ReVQ-2k dataset, featuring urban, interior, and landscape environments under different weather conditions.
• Figure 3: Distributions of low-level quantitative attributes of our proposed ReVQ-2k dataset and five existing datasets.
• Figure 4: Histograms (a) and (b) and scatter plot (c) of OA-MOS and TS-MOS from our proposed ReVQ-2k dataset. The project homepage includes separate data analyses for each resolution and display.
• Figure 5: Overview of our NR-VQA model. Given an input video $V$, our model employs two streams, (a) and (b), to extract features relevant to image quality and temporal stability, respectively. Stream (a) utilizes the FAST-VQA framework wu2023neighbourhood to normalize input data and feeds the sampled video $V'$ into a Swin-T model, which extracts features that are processed through nonlinear layers to derive the image quality score $q_a$. Stream (b) assesses temporal stability by aligning frames using motion estimation, image warping, and occlusion removal, followed by applying image differencing to compute the temporal score $q_b$. Both scores, $q_a$ and $q_b$, are integrated using an MLP to predict the overall video quality. Notably, stream (b) can be trained using TS-MOS labels to improve the accuracy of NR-VQA for rendered videos.