Learning Motion and Temporal Cues for Unsupervised Video Object Segmentation

TL;DR

MTNet addresses unsupervised video object segmentation by jointly exploiting appearance, motion, and long-range temporal context. It introduces a Bi-modal Fusion Module for efficient cross-modal feature integration, a Mixed Temporal Transformer to model inter-frame dynamics, and a Cascaded Transformer Decoder for progressive multi-level refinement, achieving state-of-the-art UVOS results and competitive VSOD performance. Two-stage training (YouTube-VOS pre-training and DAVIS-16 fine-tuning) plus a lightweight design yield strong accuracy with real-time-like inference on standard GPUs. This framework demonstrates robust performance across diverse datasets and long video sequences, highlighting the practical relevance for real-time video analysis and surveillance tasks. The work also provides insights into modality fusion and temporal modeling that can inform future multimodal video segmentation research.

Abstract

In this paper, we address the challenges in unsupervised video object segmentation (UVOS) by proposing an efficient algorithm, termed MTNet, which concurrently exploits motion and temporal cues. Unlike previous methods that focus solely on integrating appearance with motion or on modeling temporal relations, our method combines both aspects by integrating them within a unified framework. MTNet is devised by effectively merging appearance and motion features during the feature extraction process within encoders, promoting a more complementary representation. To capture the intricate long-range contextual dynamics and information embedded within videos, a temporal transformer module is introduced, facilitating efficacious inter-frame interactions throughout a video clip. Furthermore, we employ a cascade of decoders all feature levels across all feature levels to optimally exploit the derived features, aiming to generate increasingly precise segmentation masks. As a result, MTNet provides a strong and compact framework that explores both temporal and cross-modality knowledge to robustly localize and track the primary object accurately in various challenging scenarios efficiently. Extensive experiments across diverse benchmarks conclusively show that our method not only attains state-of-the-art performance in unsupervised video object segmentation but also delivers competitive results in video salient object detection. These findings highlight the method's robust versatility and its adeptness in adapting to a range of segmentation tasks. Source code is available on https://github.com/hy0523/MTNet.

Figures (9)

• Figure 1: Three distinct methodologies for unsupervised video object segmentation (UVOS). (a) The video-based approach consolidates temporal information to enhance feature representation across frames. (b) The appearance-motion-based method utilizes optical flow for motion guidance and compensation. (c) Our proposed method innovatively synthesizes both temporal and motion information within a cohesive framework, facilitating the transfer of cross-modal and cross-frame knowledge.
• Figure 2: (a) The proposed MTNet pipeline utilizes $t$ frames of images and flow maps as input to extract multi-level features. These features at each level are fused by the (b) Bi-modal Feature Fusion Module. Subsequently, the temporal modeling of high-level features are achieved through the (c) Mixed Temporal Transformer. Finally, the output masks are generated using the (d) Cascaded Transformer Decoder.
• Figure 3: Illustration of (a) Local Window MHSA and (b) Global Summarization MHSA.
• Figure 4: Visual analysis of unsupervised video object segmentation performance across a variety of video scenarios. The sequence of frames illustrates the algorithm's ability to maintain consistent object segmentation over time, highlighting its effectiveness in various contexts, from group gatherings to fast-moving vehicles and interactions in natural settings.
• Figure 5: Visual comparison of the saliency maps between our methods and state-of-the-art models. This figure illustrates the precision and clarity with which our method delineates salient objects, maintaining consistency and accuracy against ground truth (GT) benchmarks. The saliency maps demonstrate our method's superior performance in detecting and segmenting salient objects in diverse and dynamic video scenes.