Point-aware Interaction and CNN-induced Refinement Network for RGB-D Salient Object Detection

TL;DR

This work addresses RGB-D salient object detection by bridging global context modeling and local detail preservation. It proposes PICR-Net, a CNN-assisted Transformer network that uses a cross-modality point-aware interaction (CmPI) module to fuse RGB and depth features at corresponding spatial locations under global guidance, and a CNN-induced refinement (CNNR) unit at the end to recover fine details. CmPI employs masks and global saliency guidance to enable targeted cross-modality interactions, while CNNR refines the Transformer outputs using shallow CNN features from VGG16 for boundary accuracy. Extensive experiments on five RGB-D SOD datasets show competitive performance and favorable inference speed, with ablations confirming the importance of CmPI and CNNR in achieving high-quality saliency maps.

Abstract

By integrating complementary information from RGB image and depth map, the ability of salient object detection (SOD) for complex and challenging scenes can be improved. In recent years, the important role of Convolutional Neural Networks (CNNs) in feature extraction and cross-modality interaction has been fully explored, but it is still insufficient in modeling global long-range dependencies of self-modality and cross-modality. To this end, we introduce CNNs-assisted Transformer architecture and propose a novel RGB-D SOD network with Point-aware Interaction and CNN-induced Refinement (PICR-Net). On the one hand, considering the prior correlation between RGB modality and depth modality, an attention-triggered cross-modality point-aware interaction (CmPI) module is designed to explore the feature interaction of different modalities with positional constraints. On the other hand, in order to alleviate the block effect and detail destruction problems brought by the Transformer naturally, we design a CNN-induced refinement (CNNR) unit for content refinement and supplementation. Extensive experiments on five RGB-D SOD datasets show that the proposed network achieves competitive results in both quantitative and qualitative comparisons.

Figures (6)

• Figure 1: Visual comparison of representative networks with different architectures, where MVSalNet MVSalNet, VST vst and TriTransNet TriTransNet are the pure CNNs, pure Transformer, and Transformer-assisted CNNs architectures, respectively.
• Figure 2: The overall framework of the proposed PICR-Net. First, RGB image and depth image are fed to a dual-stream encoder to extract corresponding multi-level features $\{f_{r}^{i}\}_{i=1}^{4}$ and $\{f_{d}^{i}\}_{i=1}^{4}$. Subsequently, the features of the same layer are multi-dimensionally interacted through cross-modality point-aware interaction module, where the previously output saliency map $S_{i+1}$ is used to extract global guidance information. At the end of the network, the CNNR unit provides convolutional features with higher resolution and more detail from the pre-trained VGG16 model to refine and output the final high-quality saliency map $S_{out}$.
• Figure 3: The cross-modality point-aware RM in the CmPI module, where the RGB and depth features at the same spatial location and the global saliency guidance vectors from both modalities are interacted sufficiently and efficiently.
• Figure 4: Visual comparisons between our PICR-Net and SOTA methods under different challenging scenes, such as small objects (i.e., a, c and d), multiple objects (i.e., c), low contrast (i.e., d and f), low-quality depth map (i.e., b and e), and uneven lighting (i.e., g).
• Figure 5: Visual comparisons of different ablation studies.