Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks

TL;DR

The paper tackles RGB-D salient object detection in real-world human activity scenes by introducing the SIP dataset and a large-scale seven-dataset benchmark, totaling about 97K images. It presents D3Net, a simple yet effective model with a Depth Depurator Unit that filters low-quality depth maps and a three-stream feature learning module for cross-modal fusion, achieving state-of-the-art results across multiple metrics and capable of fast, real-time saliency extraction. Extensive experiments demonstrate robust gains over prior methods and provide a comprehensive online benchmark for fair comparisons, with public data and tools to support future research. Overall, this work advances RGB-D SOD towards practical mobile and real-world applications by combining a new dataset, large benchmark, and a generalizable model architecture.

Abstract

The use of RGB-D information for salient object detection has been extensively explored in recent years. However, relatively few efforts have been put towards modeling salient object detection in real-world human activity scenes with RGBD. In this work, we fill the gap by making the following contributions to RGB-D salient object detection. (1) We carefully collect a new SIP (salient person) dataset, which consists of ~1K high-resolution images that cover diverse real-world scenes from various viewpoints, poses, occlusions, illuminations, and backgrounds. (2) We conduct a large-scale (and, so far, the most comprehensive) benchmark comparing contemporary methods, which has long been missing in the field and can serve as a baseline for future research. We systematically summarize 32 popular models and evaluate 18 parts of 32 models on seven datasets containing a total of about 97K images. (3) We propose a simple general architecture, called Deep Depth-Depurator Network (D3Net). It consists of a depth depurator unit (DDU) and a three-stream feature learning module (FLM), which performs low-quality depth map filtering and cross-modal feature learning respectively. These components form a nested structure and are elaborately designed to be learned jointly. D3Net exceeds the performance of any prior contenders across all five metrics under consideration, thus serving as a strong model to advance research in this field. We also demonstrate that D3Net can be used to efficiently extract salient object masks from real scenes, enabling effective background changing application with a speed of 65fps on a single GPU. All the saliency maps, our new SIP dataset, the D3Net model, and the evaluation tools are publicly available at https://github.com/DengPingFan/D3NetBenchmark.

Figures (12)

• Figure 1: Left to right: input image, ground truth, and the corresponding depth map. The quality of the depth map from low ($1^{st}$ row), mid ($2^{nd}$ row) to high (last row). As shown in the 2$^{nd}$ row, it is difficult to recognize the boundary of the human's arm in the boundary box region. However, it is clearly visible in the depth map. The high-quality depth maps benefit the RGB-D based salient object detection task. These three examples are from the NJU2K ju2014depth, our SIP and NLPR peng2014rgbd datasets respectively.
• Figure 2: Representative subsets in our SIP. The images in SIP are grouped into eight subsets according to background objects (i.e., grass, car, barrier, road, sign, tree, flower, and other), different lighting conditions (i.e., low-light, sunny with clear object boundary) and various number of objects (i.e., 1, 2, $\geq$3).
• Figure 3: Examples of images, depth maps and annotations (i.e., object-level, instance-level) in our SIP dataset with different numbers of salient objects, object sizes, object positions, scene complexities, and lighting conditions. Note that the "RGB" and "Gray" images are captured by two different monocular cameras from short distances. Thus, the "Gray" images are slightly different from the grayscale images obtained from colorful (RGB) image. Our SIP dataset provides a new direction such as depth estimating from "RGB" and "Gray" images, and instance-level RGB-D salient object detection.
• Figure 4: (a) Distribution of normalized object center distance from image center. (b) Distribution of normalized object margin (farthest point in an object) distance from image center. (c) Distribution of normalized object size.
• Figure 5: Illustration of the proposed D$^3$Net. In the training stage (Left), the input RGB and depth images are processed with three parallel sub-networks, e.g., RgbNet, RgbdNet, and DepthNet. The three sub-networks are based on a same modified structure of Feature Pyramid Networks (FPN) (see $\S$\ref{['sec:FPN']} for details). We introduced these sub-networks to obtain three saliency maps (i.e., $S_{rgb}$, $S_{rgbd}$, and $S_{depth}$) which considered both coarse and fine details of the input. In the test phase (Right), a novel depth depurator unit (DDU) ($\S$\ref{['sec:DDU']}) is utilized for the first time in this work to explicitly discard (i.e., $S_{rgbd}$) or keep (i.e., $S_{rgbd}$) the saliency map introduced by the depth map. In the training/test phase, these components form a nested structure and are elaborately designed (e.g., gate connection in DDU) to automatically learn the salient object from the RGB image and Depth image jointly.