VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM

TL;DR

This work introduces the VideoRefer Suite to enable fine-grained spatial-temporal understanding in Video LLMs by coupling a large-scale object-level dataset (VideoRefer-700K) with a spatial-temporal encoder (VideoRefer) and a dual-branch benchmark (VideoRefer-Bench) for description and QA tasks. The model employs a versatile Spatial-Temporal Object Encoder featuring a Spatial Token Extractor and a Temporal Token Merge Module to produce object-level tokens that are interleaved with scene-level and language representations for LLM decoding. Extensive experiments show state-of-the-art performance on both video-referring tasks and general video understanding benchmarks, driven by rich object-level data and robust region-text alignment. The approach significantly advances interactive, object-centric reasoning over videos, with potential applications in surveillance, robotics, and interactive AI systems.

Abstract

Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. Besides, the lack of high-quality object-level video instruction data and a comprehensive benchmark further hinders their advancements. To tackle these challenges, we introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding, i.e., enabling perception and reasoning on any objects throughout the video. Specially, we thoroughly develop VideoRefer Suite across three essential aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent data engine to meticulously curate a large-scale, high-quality object-level video instruction dataset, termed VideoRefer-700K. Next, we present the VideoRefer model, which equips a versatile spatial-temporal object encoder to capture precise regional and sequential representations. Finally, we meticulously create a VideoRefer-Bench to comprehensively assess the spatial-temporal understanding capability of a Video LLM, evaluating it across various aspects. Extensive experiments and analyses demonstrate that our VideoRefer model not only achieves promising performance on video referring benchmarks but also facilitates general video understanding capabilities.

Figures (14)

• Figure 1: Comparisons with previous general and specialized MLLMs. Our VideoRefer excels in multiple fine-grained regional and temporal video understanding tasks, including basic video object referring, complex video relationship analysis, and video object retrieval.
• Figure 2: A multi-agent data engine for the construction of our VideoRefer-700K.
• Figure 3: Model architecture of our VideoRefer for spatial-temporal video object understanding.
• Figure 4: Exemplar visual illustration of VideoRefer-Bench.
• Figure 5: Data characteristics of VideoRefer-Bench.