Adaptive Multi-source Predictor for Zero-shot Video Object Segmentation

TL;DR

This work tackles zero-shot video object segmentation (ZVOS) by introducing an adaptive multi-source predictor that leverages RGB, depth, optical flow, and static saliency. It decomposes the problem into a static object predictor and a moving object predictor, connected by an adaptive predictor fusion (APF) network that weighs predictions by optical-flow quality to prevent failure from poor flow maps. Key innovations include the interoceptive spatial attention module (ISAM), the motion-enhanced module (MEM), and the feature purification module (FPM) for robust multi-source fusion, plus a three-stage training strategy and multi-metric supervision for APF. The method achieves state-of-the-art results on DAVIS-16, YouTube-Objects, and FBMS, and generalizes to RGB-D salient object detection and depth estimation, demonstrating versatile cross-modal capabilities and improved robustness to degraded optical-flow input.

Abstract

Static and moving objects often occur in real-life videos. Most video object segmentation methods only focus on extracting and exploiting motion cues to perceive moving objects. Once faced with the frames of static objects, the moving object predictors may predict failed results caused by uncertain motion information, such as low-quality optical flow maps. Besides, different sources such as RGB, depth, optical flow and static saliency can provide useful information about the objects. However, existing approaches only consider either the RGB or RGB and optical flow. In this paper, we propose a novel adaptive multi-source predictor for zero-shot video object segmentation (ZVOS). In the static object predictor, the RGB source is converted to depth and static saliency sources, simultaneously. In the moving object predictor, we propose the multi-source fusion structure. First, the spatial importance of each source is highlighted with the help of the interoceptive spatial attention module (ISAM). Second, the motion-enhanced module (MEM) is designed to generate pure foreground motion attention for improving the representation of static and moving features in the decoder. Furthermore, we design a feature purification module (FPM) to filter the inter-source incompatible features. By using the ISAM, MEM and FPM, the multi-source features are effectively fused. In addition, we put forward an adaptive predictor fusion network (APF) to evaluate the quality of the optical flow map and fuse the predictions from the static object predictor and the moving object predictor in order to prevent over-reliance on the failed results caused by low-quality optical flow maps. Experiments show that the proposed model outperforms the state-of-the-art methods on three challenging ZVOS benchmarks. And, the static object predictor precisely predicts a high-quality depth map and static saliency map at the same time.

Figures (14)

• Figure 1: Failure cases in AMCNet AMCNet and FSNet FSNet. These two respective sequences ($breakdance$ and $cars$) are selected from the DAVIS$_{16}$davis16 and FBMS FBMS dataset, respectively. These difficult samples have obvious interference from low-quality optical flow maps.
• Figure 2: Some pairs of RGB and optical flow maps in static and moving frames. The static frames $cat$ and $cow$ are randomly selected from the Youtube-Objects youtube-objects. The moving frames $bmx-bumps$ and $soccerball$ are randomly selected from the DAVIS$_{16}$davis16.
• Figure 3: Visual results of different sources. Samples $drift-chicane$ and $aeroplane$ are randomly selected from the DAVIS$_{16}$davis16 and Youtube-Objects youtube-objects, respectively.
• Figure 4: Network pipeline of the ZVOS task. It consists of three stages: static object predictor, moving object predictor and adaptive predictor fusion. The first stage network has two-fold function: 1) It used to generate features of RGB, depth and static saliency for the second stage. 2) It can produce static salient object segmentation (SOS). The second stage network achieves multi-source fusion and yields moving object segmentation prediction (MOS). The third stage network fuses the predictions from the static object predictor and moving object predictor as the final output.
• Figure 5: Illustration of the static predictor network.