Omnidirectional Image Quality Captioning: A Large-scale Database and A New Model

TL;DR

This work tackles the challenge of omnidirectional image quality assessment under heterogeneous distortions by introducing the OIQ-10K database and a novel caption-based quality model, IQCaption360. The approach leverages a transformer-based viewport backbone with multitask learning to predict distortion type and a quality score, which are then composed into a textual quality caption. Empirical results show substantial gains over state-of-the-art methods and good cross-dataset generalization, while maintaining real-time inference speed. The dataset and model advance practical OIQA for VR applications and open avenues for integrating language models to produce richer quality descriptions.

Abstract

The fast growing application of omnidirectional images calls for effective approaches for omnidirectional image quality assessment (OIQA). Existing OIQA methods have been developed and tested on homogeneously distorted omnidirectional images, but it is hard to transfer their success directly to the heterogeneously distorted omnidirectional images. In this paper, we conduct the largest study so far on OIQA, where we establish a large-scale database called OIQ-10K containing 10,000 omnidirectional images with both homogeneous and heterogeneous distortions. A comprehensive psychophysical study is elaborated to collect human opinions for each omnidirectional image, together with the spatial distributions (within local regions or globally) of distortions, and the head and eye movements of the subjects. Furthermore, we propose a novel multitask-derived adaptive feature-tailoring OIQA model named IQCaption360, which is capable of generating a quality caption for an omnidirectional image in a manner of textual template. Extensive experiments demonstrate the effectiveness of IQCaption360, which outperforms state-of-the-art methods by a significant margin on the proposed OIQ-10K database. The OIQ-10K database and the related source codes are available at https://github.com/WenJuing/IQCaption360.

Figures (9)

• Figure 1: Visual examples represented by quality value and quality caption. QV represents Quality Value, and QC represents Quality Caption.
• Figure 2: The scatter plots of Spatial Information (SI) and Colorfulness (CF) of existing OIQA databases and the proposed OIQ-10K database.
• Figure 3: Visualization of omnidirectional images with different distortion situations in the proposed OIQ-10K database. The distorted region(s) of the visual examples in (b) and (c) are marked in red for better visual presentation.
• Figure 4: Statistics in OIQ-10K database: (a) The distribution of MOSs; (b) The scatter plot of the mean ($\mu$) and variance ($\sigma^2$ ) of the images with different degradation situations; (c) The box plot of MOSs in the CdistR1 and CdistR2 situations.
• Figure 5: The box plot of MOSs under different viewing conditions.