Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation

TL;DR

This work addresses the gap in referring video object segmentation for long-form videos by introducing Long-RVOS, a minute-scale benchmark with 2,193 videos (mean duration 60.3s), 6,703 objects, and 24,689 descriptions across Static, Dynamic, and Hybrid types. It introduces two metrics, $tIoU$ and $vIoU$, to comprehensively evaluate temporal and spatiotemporal consistency alongside traditional frame-wise $ ext{J} abla ext{F}$, and benchmarks seven SOTA RVOS methods on this challenging dataset. To tackle long-term reasoning efficiently, the authors propose ReferMo, a clip-based baseline that uses MPEG-4 motion vectors to expand the temporal receptive field and a local-to-global architecture for cross-clip interactions, predicting masks only on keyframes and propagating via SAM2. Results show that existing methods struggle with long videos, while ReferMo achieves notable improvements, offering a practical baseline and a foundation for advancing robust long-form RVOS research.

Abstract

Referring video object segmentation (RVOS) aims to identify, track and segment the objects in a video based on language descriptions, which has received great attention in recent years. However, existing datasets remain focus on short video clips within several seconds, with salient objects visible in most frames. To advance the task towards more practical scenarios, we introduce \textbf{Long-RVOS}, a large-scale benchmark for long-term referring video object segmentation. Long-RVOS contains 2,000+ videos of an average duration exceeding 60 seconds, covering a variety of objects that undergo occlusion, disappearance-reappearance and shot changing. The objects are manually annotated with three different types of descriptions to individually evaluate the understanding of static attributes, motion patterns and spatiotemporal relationships. Moreover, unlike previous benchmarks that rely solely on the per-frame spatial evaluation, we introduce two new metrics to assess the temporal and spatiotemporal consistency. We benchmark 6 state-of-the-art methods on Long-RVOS. The results show that current approaches struggle severely with the long-video challenges. To address this, we further propose ReferMo, a promising baseline method that integrates motion information to expand the temporal receptive field, and employs a local-to-global architecture to capture both short-term dynamics and long-term dependencies. Despite simplicity, ReferMo achieves significant improvements over current methods in long-term scenarios. We hope that Long-RVOS and our baseline can drive future RVOS research towards tackling more realistic and long-form videos.

Figures (8)

• Figure 1: Duration comparison of current RVOS datasets. The circle size indicates the number of frames.
• Figure 2: Examples from Long-RVOS dataset, with frame indices displayed in the upper left, and selected objects masked in orange $\blacksquare$. Long-RVOS contains extensive long-term videos, where the objects always undergo occlusion, disappearance-reappearance and shot changing. In addition, the objects are annotated with three different types descriptions: Static, Dynamic and Hybrid.
• Figure 3: Frequency distribution of the Top-50 object categories.
• Figure 4: Overview of objects and scenes in Long-RVOS.
• Figure 5: Representative statistics of Long-RVOS.