Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentation

TL;DR

This work addresses Referring Video Object Segmentation under a training-free, zero-shot paradigm. It introduces Refer-Agent, a collaborative multi-agent system that decomposes RVOS into four reasoning stages and augments it with a Chain-of-Reflection to mitigate MLLM hallucinations via Existence and Consistency reflections. Key innovations include a Coarse-to-Fine frame selection strategy, a Dynamic Focus Layout to emphasize keyframes, and a feedback-driven reasoning-reflection loop. Empirical results across five RVOS benchmarks show state-of-the-art performance without fine-tuning and demonstrate robust integration with different MLLMs, highlighting practical scalability and robustness for real-time and evolving systems.

Abstract

Referring Video Object Segmentation (RVOS) aims to segment objects in videos based on textual queries. Current methods mainly rely on large-scale supervised fine-tuning (SFT) of Multi-modal Large Language Models (MLLMs). However, this paradigm suffers from heavy data dependence and limited scalability against the rapid evolution of MLLMs. Although recent zero-shot approaches offer a flexible alternative, their performance remains significantly behind SFT-based methods, due to the straightforward workflow designs. To address these limitations, we propose \textbf{Refer-Agent}, a collaborative multi-agent system with alternating reasoning-reflection mechanisms. This system decomposes RVOS into step-by-step reasoning process. During reasoning, we introduce a Coarse-to-Fine frame selection strategy to ensure the frame diversity and textual relevance, along with a Dynamic Focus Layout that adaptively adjusts the agent's visual focus. Furthermore, we propose a Chain-of-Reflection mechanism, which employs a Questioner-Responder pair to generate a self-reflection chain, enabling the system to verify intermediate results and generates feedback for next-round reasoning refinement. Extensive experiments on five challenging benchmarks demonstrate that Refer-Agent significantly outperforms state-of-the-art methods, including both SFT-based models and zero-shot approaches. Moreover, Refer-Agent is flexible and enables fast integration of new MLLMs without any additional fine-tuning costs. Code will be released.

Figures (5)

• Figure 1: Comparison with SOTAs. Without any fine-tuning, our Refer-Agent achieves the best performances across five RVOS datasets and outperforms all previous state-of-the-art methods, including both zero-shot approaches and SFT-based models.
• Figure 2: (a) Overview of the Refer-Agent system. The primary pipeline (blue) performs step-by-step reasoning to achieve object analysis and segmentation, while a Two-stage Chain-of-Reflection mechanism (orange) is further integrated to verify and refine the intermediate results. By alternating between reasoning and reflection, our Refer-Agent can produce robust RVOS predictions. (b) Illustration of key components. Specially, the Chain-of-Reflection mechanism, comprising Existence Reflection and Consistency Reflection, employs a Questioner-Responder pair to verify intermediate results, identify errors, and provide feedback for next-round reasoning refinement.
• Figure 3: Illustration of Dynamic Focus Layout. The number within each block denotes the frame index.
• Figure 4: Visualization of Refer-Agent's alternating reasoning-reflection process. The data flow is highlighted with yellow arrows. Left: A case of correcting frame selection via Existence Reflection. Right: A case of correcting object identification via Consistency Reflection.
• Figure 5: Qualitative results comparing our Refer-Agent with AL-Ref-SAM2 and GLUS. In the first sample, Refer-Agent correctly segments the small and motionless monkey, while competitors mistakenly segment distant zebras. In the second sample, it accurately segments the black car, whereas AL-Ref-SAM2 and GLUS incorrectly segment the white car and the yellow truck, respectively.