Deforming Videos to Masks: Flow Matching for Referring Video Segmentation

TL;DR

FlowRVS reframes RVOS as a text-conditioned continuous flow that deterministically deforms a video latent into its target mask, driven by a velocity field learned through flow matching. By treating the language query as a critical selector and introducing boundary-biased sampling, start-point augmentation, and direct video injection, the method effectively bridges generative priors and discriminative segmentation. It leverages a pretrained text-to-video model (Wan 2.1) with frozen encoders and a fine-tuned VAE decoder, achieving state-of-the-art results on MeViS and strong zero-shot generalization on Ref-DAVIS17. The work demonstrates that modeling video-to-mask transformation as a guided deformation process yields superior temporal coherence and language grounding, with broad implications for end-to-end, multi-modal video understanding.

Abstract

Referring Video Object Segmentation (RVOS) requires segmenting specific objects in a video guided by a natural language description. The core challenge of RVOS is to anchor abstract linguistic concepts onto a specific set of pixels and continuously segment them through the complex dynamics of a video. Faced with this difficulty, prior work has often decomposed the task into a pragmatic `locate-then-segment' pipeline. However, this cascaded design creates an information bottleneck by simplifying semantics into coarse geometric prompts (e.g, point), and struggles to maintain temporal consistency as the segmenting process is often decoupled from the initial language grounding. To overcome these fundamental limitations, we propose FlowRVS, a novel framework that reconceptualizes RVOS as a conditional continuous flow problem. This allows us to harness the inherent strengths of pretrained T2V models, fine-grained pixel control, text-video semantic alignment, and temporal coherence. Instead of conventional generating from noise to mask or directly predicting mask, we reformulate the task by learning a direct, language-guided deformation from a video's holistic representation to its target mask. Our one-stage, generative approach achieves new state-of-the-art results across all major RVOS benchmarks. Specifically, achieving a $\mathcal{J}\&\mathcal{F}$ of 51.1 in MeViS (+1.6 over prior SOTA) and 73.3 in the zero shot Ref-DAVIS17 (+2.7), demonstrating the significant potential of modeling video understanding tasks as continuous deformation processes.

Figures (7)

• Figure 1: FlowRVS reformulates RVOS as a text-conditioned continuous flow, learning a velocity field via Flow Matching stabilized by boundary-biased time sampling in latent space. During inference, an ODE solver uses this field to deterministically deform the video latent to the target mask, this video to mask paradigm superior to noise-based or one-step prediction approaches.
• Figure 2: Repurposing a generative process for a discriminative task. Unlike standard T2V generation which maps noise to diverse videos (left), our method maps a complex video to a single mask (right). This transforms the process into a deterministic, convergent task where the text query is the crucial element that selects the precise target from the visual input (e.g., distinguishing the 'smaller' from the 'bigger' monkey).
• Figure 3: Qualitative comparison on challenging temporal and linguistic reasoning. Prior paradigms struggle: VD-IT produces temporally unstable masks due to its frame-wise decoder, while ReferDINO fails to interpret long-range descriptions. Our method, FlowRVS, demonstrates superior temporal coherence and language grounding by leveraging an end-to-end generative process.
• Figure 4: Visualization of VAE reconstruction results
• Figure 5: visualization example of FlowRVS results on MeViS-Valid-u.