Referring Video Object Segmentation via Language-aligned Track Selection

TL;DR

This work tackles RVOS by using SAM2 object tokens as compact video-level representations and a lightweight language-aligned track selection module to bridge vision-language gaps. It introduces an IoU-based pseudo-labeling strategy to supervise the alignment while keeping SAM2 frozen, and trains a small set of parameters on a single GPU. The method achieves state-of-the-art results on the MeViS dataset (e.g., $J$ and $F$ scores of 48.6 with 32.9M trainable parameters) and demonstrates strong generalization in zero-shot and cross-dataset settings, including Ref-YouTube-VOS and Ref-DAVIS. This approach offers efficient, robust RVOS with improved multi-modal alignment and motion understanding, expanding practical applicability in interactive video tasks.

Abstract

Referring video object segmentation (RVOS) requires tracking and segmenting an object throughout a video according to a given natural language expression, demanding both complex motion understanding and the alignment of visual representations with language descriptions. Given these challenges, the recently proposed Segment Anything Model 2 (SAM2) emerges as a potential candidate due to its ability to generate coherent segmentation mask tracks across video frames, and provide an inherent spatio-temporal objectness in its object token representations. In this paper, we introduce SOLA (Selection by Object Language Alignment), a novel framework that leverages SAM2 object tokens as compact video-level object representations, which are aligned with language features through a lightweight track selection module. To effectively facilitate this alignment, we propose an IoU-based pseudo-labeling strategy, which bridges the modality gap between SAM2 representations with language features. Extensive experiments show that SOLA achieves state-of-the-art performance on the MeViS dataset and demonstrate that SOLA offers an effective solution for RVOS. Our project page is available at: https://cvlab-kaist.github.io/SOLA.

Figures (11)

• Figure 1: Teaser. Our method effectively bridges the modality gap by aligning the features obtained from fully frozen uni-modal encoders: the video segmentation model such as SAM2 ravi2024sam2 and the text encoder such as RoBERTa DBLP:journals/corr/abs-1907-11692. By directly leveraging the token representations, our approach achieves lightweight multi-modal alignment while significantly reducing the number of trainable parameters.
• Figure 2: Overall pipeline of the proposed SOLA framework. Our method selects the correct object mask track among candidates via a language-aligned track selection module. We first generate candidate mask tracks and corresponding object tokens from the fully frozen SAM2. These tokens are then aligned with language expressions, producing alignment scores that indicate selection probabilities. Mask tracks with scores above a predefined threshold are selected and merged into the final binary segmentation mask. By leveraging precomputed object tokens from SAM2, our approach minimizes trainable parameters, enabling efficient training on a single GPU.
• Figure 3: Architecture of the language-aligned track selection module, which takes object tokens and text tokens as inputs, aligning these representations to effectively capture object dynamics.
• Figure 4: Spatial and motion information in object tokens. The bold line represents the mean similarity, while the shaded region indicates the variance. The results show a certain correlation: as the mIoU between mask tracks increases, the similarity between their associated tokens also rises nearly proportionally. This tendency suggests that object tokens inherently capture spatial information, implicitly encoding object motions over time.
• Figure 5: Qualitative results of our model on MeViS. SOLA shows its ability to understand complex language expressions.