ViLLa: Video Reasoning Segmentation with Large Language Model

TL;DR

ViLLa addresses the challenge of video reasoning segmentation in complex real-world videos by introducing three task-specific modules that enable selective, context-aware reasoning over time. The Key Segment Extractor identifies informative segments, the Context Synthesizer fuses visual cues into text embeddings, and the Hierarchical Temporal Synchronizer merges frame- and video-level tokens to produce coherent masks across time. The authors also introduce VideoReasonSeg, a 3k-video benchmark with 15k object-instruction pairs generated via GPT-4V, designed to evaluate reasoning and segmentation in diverse scenarios. Across VideoReasonSeg and several referring VOS benchmarks, ViLLa achieves state-of-the-art results, demonstrating strong temporal reasoning and segmentation capabilities and providing code and data to advance research in video-based multimodal reasoning.

Abstract

Recent efforts in video reasoning segmentation (VRS) integrate large language models (LLMs) with perception models to localize and track objects via textual instructions, achieving barely satisfactory results in simple scenarios. However, they struggled to discriminate and deduce the objects from user queries in more real-world scenes featured by long durations, multiple objects, rapid motion, and heavy occlusions. In this work, we analyze the underlying causes of these limitations, and present ViLLa: Video reasoning segmentation with Large Language Model. Remarkably, our ViLLa manages to tackle these challenges through multiple core innovations: (1) a context synthesizer that dynamically encodes the user intent with video contexts for accurate reasoning, resolving ambiguities in complex queries, and (2) a hierarchical temporal synchronizer that disentangles multi-object interactions across complex temporal scenarios by modelling multi-object interactions at local and global temporal scales. To enable efficient processing of long videos, ViLLa incorporates (3) a key segment sampler that adaptively partitions long videos into shorter but semantically dense segments for less redundancy. What's more, to promote research in this unexplored area, we construct a VRS benchmark, VideoReasonSeg, featuring different complex scenarios. Our model also exhibits impressive state-of-the-art results on VideoReasonSeg, Ref-YouTube-VOS, Ref-DAVIS17, MeViS, and ReVOS. Both quantitative and qualitative experiments demonstrate that our method effectively enhances video reasoning segmentation capabilities for multimodal LLMs. The code and dataset will be available at https://github.com/rkzheng99/ViLLa.

Figures (7)

• Figure 1: Our ViLLa is an effective and efficient LMM capable of segmenting and tracking: (a) multiple objects with rapid motion; (b) objects in crowded scenes; (c) objects in long videos with occlusions.
• Figure 2: Overall framework of ViLLa. Given the input video frames and user input text query, the Key Segment Extractor is proposed to select the most query-relevant video segments from the video. Then, the Context Synthesizer aggregates text-related visual cues from the visual features to the text embeddings and selects the relevant visual features. The large language model generates text output and segmentation tokens with the input of visual features, visually enriched text embeddings, and pre-defined segmentation tokens. Finally, the segmentation tokens are fed to the Hierarchical Temporal Synchronizer to produce the final output segmentation mask tracklets.
• Figure 3: Qualitative Comparisons between ViLLa and VISA. Compared to VISA, our ViLLa successfully segments the sheep, which demonstrates better discrimination, temporal consistency, and conversation quality.
• Figure 4: GPT-4V data generation pipeline. The right part shows an example of how reasoning segmentation data and multiple choices are generated. The input prompt includes certain rules and the position as well as time localizations to instruct GPT-4V into generating more effective data samples.
• Figure 5: GPT-4V data generation samples. The part shows further samples of the generated questions and the multiple choices. The two types of questions can help us better evaluate the model's performance in video reasoning at both pixel-level and video-level.