SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation

TL;DR

SAMWISE addresses the challenge of streaming referring video segmentation by enhancing the SAM2 model with language understanding and explicit temporal modeling. It introduces the Cross-Modal Temporal Adapter to fuse visual and textual information and capture temporal evolution, plus a Lightweight Conditional Memory Encoder to correct tracking bias as new, caption-aligned objects appear. The approach preserves SAM2 weights and adds less than 5 million trainable parameters, delivering state-of-the-art results on MeViS, Ref-Youtube-VOS, and Ref-DAVIS without full-video offline processing. This work enables efficient, context-aware RVOS in streaming scenarios, handling long-term context, occlusions, and disambiguation with a compact, end-to-end model. The practical impact lies in enabling robust real-time RVOS for streaming video applications without heavy external models or fine-tuning of foundational segmentation weights.

Abstract

Referring Video Object Segmentation (RVOS) relies on natural language expressions to segment an object in a video clip. Existing methods restrict reasoning either to independent short clips, losing global context, or process the entire video offline, impairing their application in a streaming fashion. In this work, we aim to surpass these limitations and design an RVOS method capable of effectively operating in streaming-like scenarios while retaining contextual information from past frames. We build upon the Segment-Anything 2 (SAM2) model, that provides robust segmentation and tracking capabilities and is naturally suited for streaming processing. We make SAM2 wiser, by empowering it with natural language understanding and explicit temporal modeling at the feature extraction stage, without fine-tuning its weights, and without outsourcing modality interaction to external models. To this end, we introduce a novel adapter module that injects temporal information and multi-modal cues in the feature extraction process. We further reveal the phenomenon of tracking bias in SAM2 and propose a learnable module to adjust its tracking focus when the current frame features suggest a new object more aligned with the caption. Our proposed method, SAMWISE, achieves state-of-the-art across various benchmarks, by adding a negligible overhead of less than 5 M parameters. Code is available at https://github.com/ClaudiaCuttano/SAMWISE .

Figures (10)

• Figure 1: SAMWISE. Our approach infuses knowledge about natural language in the Segment-Anything 2 model, adding explicit temporal cues in the feature extraction for the task of streaming-based Referring Video Segmentation (RVOS). We use a learnable mechanism to mitigate the so-called tracking bias, i.e.SAM2 tendency to overlook a correct object once it becomes identifiable, due to its ongoing tracking of a different object. Our design enables effective streaming processing for RVOS, exploiting the memory from previous frames to propagate past context. The figure shows an example where the target object is not present in the first frame, leading SAM2 to start tracking the wrong one. Afterwards, when the correct object appears, our learnable correction mechanisms guides SAM2 to switch its tracking focus. By adding in its features the notion of temporal evolution, the model is able to recognize that the new object is more aligned with the provided textual query. Finally, we exploit SAM2 tracking skills and robustness to occlusions to keep following the object.
• Figure 2: Overview of SAMWISE. We build on a frozen SAM2 and a frozen Text Encoder to segment images in video given a textual description. We incorporate the Cross-Modal Temporal Adapter (CMT) into the text and visual encoders at every intermediate layer $k$ to model temporal dynamics within visual features while contaminating each modality with the other. Then, we extract the [CLS] and verb embeddings, namely Contextual and Motion prompts, from the adapted textual features and project them through a learnable MLP. The final embedding is used to prompt the Mask Decoder, which outputs the segmentation mask. Finally, the Conditional Memory Encoder detects when a new candidate object, aligned with the caption, appears in the frame, enabling SAM2 to dynamically refocus its tracking.
• Figure 3: Architecture of our Cross Modal Temporal (CMT) Adapter, made up of Hierarchical Selective Attention (HSA) to model temporal cues, a Visual-to-Text Attention (VTA) and Text-to-Visual Attention (VTA) modules.
• Figure 4: Scheme of our Hierarchical Selective Attention (HSA), modeling temporal evolution of features in our adapter.
• Figure 5: Cross Modal Temporal Adapter: we show via PCA that our CMT provides contextualized visual features based on the given textual prompt, compared to SAM2 original ones.