GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation

TL;DR

GLUS tackles RefVOS by unifying global and local reasoning within a single multimodal LLM, achieved by separating frames into sparse context frames for global cues and continuous query frames for local tracking. It integrates an end-to-end memory bank with a pre-trained VOS module, and introduces an object-level contrastive loss and a self-refined key-frame selector to enhance frame relevance and object discrimination within the limited context window. The approach yields state-of-the-art results on MeViS and strong performance on Ref-Youtube-VOS, demonstrating the practicality of a simple, unified baseline for video-language reasoning. Together, these components offer a scalable pathway to robust video segmentation under language guidance with reduced reliance on external VOS systems.

Abstract

This paper proposes a novel framework utilizing multi-modal large language models (MLLMs) for referring video object segmentation (RefVOS). Previous MLLM-based methods commonly struggle with the dilemma between "Ref" and "VOS": they either specialize in understanding a few key frames (global reasoning) or tracking objects on continuous frames (local reasoning), and rely on external VOS or frame selectors to mitigate the other end of the challenge. However, our framework GLUS shows that global and local consistency can be unified into a single video segmentation MLLM: a set of sparse "context frames" provides global information, while a stream of continuous "query frames" conducts local object tracking. This is further supported by jointly training the MLLM with a pre-trained VOS memory bank to simultaneously digest short-range and long-range temporal information. To improve the information efficiency within the limited context window of MLLMs, we introduce object contrastive learning to distinguish hard false-positive objects and a self-refined framework to identify crucial frames and perform propagation. By collectively integrating these insights, our GLUS delivers a simple yet effective baseline, achieving new state-of-the-art for MLLMs on the MeViS and Ref-Youtube-VOS benchmark. Our project page is at https://glus-video.github.io/.

Figures (4)

• Figure 1: Overview of GLUS. (a) We present a representative RefVOS example where the referring expression only covers part of the frames (the orange arrows point to the ground truth). Therefore, RefVOS requires both global reasoning for finding the target objects in frames without the referred motions and local reasoning for predicting temporally consistent masks. (b) We show green and red masks for correct and wrong segmentation. Existing multimodal large language models (MLLMs) focus on either global or local reasoning. However, global-only shows fluctuation in local "VOS," while local-only "Ref" to wrong objects without overall video contexts when the referring is not happening. Instead, our unified global-local reasoning shows consistent and correct predictions. (c) GLUS provides a simple yet effective baseline that unifies global and local reasoning through both global and local frame sampling and end-to-end memory modules.
• Figure 2: Overview of our GLUS. The yellow arrows mark the flow. (1) Sec. \ref{['sec:global-local-segmentor']}. Beginning from a video and referring, we design context frames and query frames to unify the distinct global and local reasoning into a single MLLM. The $\left<\mathtt{SEG}\right>$ tokens represent the target object. (2) Sec. \ref{['sec:memory_bank']}. GLUS end-to-end integrates pre-trained VOS memory modules to enhance temporal reasoning and decouple the reliance of existing models on VOS post-processing. The memory-enhanced decoder decodes $\left<\mathtt{SEG}\right>$ tokens into masks. (3) Sec. \ref{['sec:contrastive-loss']}. The $\left<\mathtt{SEG}\right>$ tokens are further supervised to better distinguish the different objects. (4) Sec. \ref{['sec:selector']}. The accuracy of $\left<\mathtt{SEG}\right>$ can be used for finetuning a plug-and-play key frame selector to further enhance GLUS's inference-time ability.
• Figure 3: Qualitative comparisons on our key designs. "MB": End-to-end memory bank, "OC": Object contrastive loss. Our memory bank improves the temporal consistency of segmentation, and our object contrastive loss encourages more precise localization of objects.
• Figure 4: Curves of MeViS valid_u performance ($\mathcal{J}$&$\mathcal{F}$) with distinct training steps. The figure clearly demonstrates noticeable overfitting in the model. "GLU": Global-local unification, "MB": End-to-end memory bank, "OC": Object contrastive loss.