VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding

TL;DR

VideoLoom tackles the challenge of unified joint spatial-temporal video understanding by introducing LoomData-8.7k, a richly annotated dataset with temporally grounded and spatially localized captions, and LoomBench for comprehensive evaluation. The model combines SlowFast visual tokens with an MLLM-SAM2 architecture to process both temporal dynamics and high-resolution spatial details within a single framework, enabling text outputs with timestamps and segmentation masks. Empirical results show VideoLoom achieves state-of-the-art or competitive performance across temporal and spatial benchmarks, and LoomBench demonstrates clear benefits for joint tasks. The work offers a scalable, end-to-end approach to multimodal video understanding that advances fine-grained reasoning over complex events in video data.

Abstract

This paper presents VideoLoom, a unified Video Large Language Model (Video LLM) for joint spatial-temporal understanding. To facilitate the development of fine-grained spatial and temporal localization capabilities, we curate LoomData-8.7k, a human-centric video dataset with temporally grounded and spatially localized captions. With this, VideoLoom achieves state-of-the-art or highly competitive performance across a variety of spatial and temporal benchmarks (e.g., 63.1 J&F on ReVOS for referring video object segmentation, and 48.3 R1@0.7 on Charades-STA for temporal grounding). In addition, we introduce LoomBench, a novel benchmark consisting of temporal, spatial, and compositional video-question pairs, enabling a comprehensive evaluation of Video LLMs from diverse aspects. Collectively, these contributions offer a universal and effective suite for joint spatial-temporal video understanding, setting a new standard in multimodal intelligence.

Figures (12)

• Figure 1: Illustration of the designed data annotation pipeline, comprising four stages: shot partition, spatial mask annotation, shot merging, and temporal action annotation. During spatial mask annotation, main characters and their complete tracklets are identified. In temporal action annotation, actions of characters are temporally grounded with visual prompts.
• Figure 2: Overview of VideoLoom Architecture. Two key designs are: (a) MLLM-SAM2 Architecture, where MLLM and SAM2 are connected via a [SEG] token, unifying temporal understanding and spatial perception. (b) SlowFast Tokens, where input videos are encoded as SlowFast visual tokens to model spatial-temporal representations.
• Figure 3: Visualization of the QA pairs in LoomBench. Three types of QA are shown: When targets the action timestamps given a query and the whole video, Where targets the person masklet given a query and a certain video segment, while Combined directly targets the tracklet segment corresponding to the query.
• Figure 4: Visualization of the predictions by VideoLoom on different spatial-temporal understanding tasks. From top to down, we show the visualization results of video temporal grounding on Charades-STA gao2017tall, referring VOS on MeVIS ding2023mevis, and reasoning VOS on ReVOS yan2024visa.
• Figure 5: Visualization of VideoLoom on LoomBench for When, Where, and Combined questions.