No-Reference Point Cloud Quality Assessment via Graph Convolutional Network

TL;DR

A novel no-reference PCQA method is proposed by using a graph convolutional network (GCN) to characterize the mutual dependencies of multi-view 2D projected image contents to achieve superior performance than state-of-the-art quality assessment metrics.

Abstract

Three-dimensional (3D) point cloud, as an emerging visual media format, is increasingly favored by consumers as it can provide more realistic visual information than two-dimensional (2D) data. Similar to 2D plane images and videos, point clouds inevitably suffer from quality degradation and information loss through multimedia communication systems. Therefore, automatic point cloud quality assessment (PCQA) is of critical importance. In this work, we propose a novel no-reference PCQA method by using a graph convolutional network (GCN) to characterize the mutual dependencies of multi-view 2D projected image contents. The proposed GCN-based PCQA (GC-PCQA) method contains three modules, i.e., multi-view projection, graph construction, and GCN-based quality prediction. First, multi-view projection is performed on the test point cloud to obtain a set of horizontally and vertically projected images. Then, a perception-consistent graph is constructed based on the spatial relations among different projected images. Finally, reasoning on the constructed graph is performed by GCN to characterize the mutual dependencies and interactions between different projected images, and aggregate feature information of multi-view projected images for final quality prediction. Experimental results on two publicly available benchmark databases show that our proposed GC-PCQA can achieve superior performance than state-of-the-art quality assessment metrics. The code will be available at: https://github.com/chenwuwq/GC-PCQA.

Figures (7)

• Figure 1: We simulate the perceptual process of HVS to perform multi-view projection on the 3D point cloud and build the graph based on the projected images. The red node in graph convolution represents the central node of the current convolution process, and the red line represents the adjacency relationship. The central node will constantly exchange information with neighboring nodes to aggregate feature information from neighbors.
• Figure 2: Framework of the proposed GC-PCQA method. It is mainly composed of three parts: multi-view projection, graph construction, and GCN-based quality prediction. Firstly, multi-view projection is performed on the point cloud to obtain a set of horizontally and vertically projected 2D images. All these projected images are fed into a pre-trained backbone and an attention block for attentive feature extraction. Secondly, a multi-level fusion of attentive feature maps is carried out through the multi-level conversion module, and graph construction is performed according to spatial relations among different projected images. Thirdly, reasoning on the constructed graph is performed by GCN to model the mutual dependencies between nodes and generate more effective feature representations. Finally, multi-level feature fusion is carried out on the feature representations obtained by two GCNs to predict the final quality score.
• Figure 3: The attention block is used on the feature map extracted by the pre-trained backbone to obtain the attentive feature map. The upper part extracts spatial attention for the input feature map, and the lower part extracts channel attention. $H$: image height, $W$: image width, $C$: image channel.
• Figure 4: Scatter plot between objective prediction scores and MOS for the top ten PCQA metrics in the experiment. The X-axis is the objective prediction score of the PCQA metric, and the Y-axis is the corresponding MOS. The first ten figures are the results on SJTU database, from top left to bottom right are $PSNR_Y$mekuria2017per, 3D-NSSzhang2022no, IT-PCQAyang2022no, ResSCNNliu2023point, VIFPsheikh2006image, SSIMwang2004image, IW-SSIMwang2010info, MS-SSIMwang2003multi, GraphSIMdiniz2020local and our proposed method. The last ten figures are the results on the WPC database, from top left to bottom right are PCQMmeynet2020pcqm, $PSNR_Y$mekuria2017per, GraphSIMdiniz2020local, 3D-NSSzhang2022no, PQANetliu2021pqa, SSIMwang2004image, MS-SSIMwang2003multi, VIFPsheikh2006image, IW-SSIMwang2010info and our proposed method. R-Square score, 95% confidence interval, and fitted curve are calculated for each scatter plot.
• Figure 5: Performance comparison results of NR method in different partition test databases of SJTU and WPC. The SJTU database uses 9-fold cross-validation, while the WPC database uses 5-fold cross-validation.