ViDSOD-100: A New Dataset and a Baseline Model for RGB-D Video Salient Object Detection

TL;DR

This work proposes a new baseline model, named attentive triple-fusion network (ATF-Net), for RGB-D video salient object detection, which aggregates the appearance information from an input RGB image, spatio-temporal information from an estimated motion map, and the geometry information from the depth map by devising three modality-specific branches and a multi-modality integration branch.

Abstract

With the rapid development of depth sensor, more and more RGB-D videos could be obtained. Identifying the foreground in RGB-D videos is a fundamental and important task. However, the existing salient object detection (SOD) works only focus on either static RGB-D images or RGB videos, ignoring the collaborating of RGB-D and video information. In this paper, we first collect a new annotated RGB-D video SOD (ViDSOD-100) dataset, which contains 100 videos within a total of 9,362 frames, acquired from diverse natural scenes. All the frames in each video are manually annotated to a high-quality saliency annotation. Moreover, we propose a new baseline model, named attentive triple-fusion network (ATF-Net), for RGB-D video salient object detection. Our method aggregates the appearance information from an input RGB image, spatio-temporal information from an estimated motion map, and the geometry information from the depth map by devising three modality-specific branches and a multi-modality integration branch. The modality-specific branches extract the representation of different inputs, while the multi-modality integration branch combines the multi-level modality-specific features by introducing the encoder feature aggregation (MEA) modules and decoder feature aggregation (MDA) modules. The experimental findings conducted on both our newly introduced ViDSOD-100 dataset and the well-established DAVSOD dataset highlight the superior performance of the proposed ATF-Net. This performance enhancement is demonstrated both quantitatively and qualitatively, surpassing the capabilities of current state-of-the-art techniques across various domains, including RGB-D saliency detection, video saliency detection, and video object segmentation. Our data and our code are available at github.com/jhl-Det/RGBD_Video_SOD.

Figures (8)

• Figure 1: The examples of proposed RGB-D video salient object detection dataset (ViDSOD-100) with pixel-level annotations.
• Figure 2: Saliency shift example of our dataset (ViDSOD-100).
• Figure 3: Statistics of the proposed ViDSOD-100. (a) Salient object categories. (b) Distribution of salient object ratios across our ViDSOD-100 dataset and three other expansive VOD datasets. (c) Co-emergence of different categories in (a).
• Figure 4: Center bias of our ViDSOD-100 and existing SOD datasets.
• Figure 5: The schematic illustration of our ATF-Net for RGB-D video saliency detection. Our ATF-Net contains three modality-specific branches and one multi-modality integration branch to fuse the appearance, temporal, and geometry information from the input RGB image, an estimated flow map, and depth image. Moreover, five modality-specific encoder feature aggregation (MEA) modules and four modality-specific decoder feature aggregation (MDA) modules are devised to integrate multi-level features at the encoders and the decoders of three modality-specific branches. "C" denotes the concatenation operation. "$BConv$" represents a sequential operation containing a $3\times3$ convolution layer, a batch normalization, and a $ReLU$ activation function. "$1*1 \ Conv$" is a $1\times1$ convolution layer.