VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding

TL;DR

VideoExpert addresses the challenge of temporal-sensitive video understanding by decoupling temporal grounding from content generation through two parallel experts: a Temporal Expert for grounding and a Spatial Expert for content generation. A Spatial Compress module reduces patch-token redundancy while preserving detail, enabling efficient processing within a single multimodal LLM. Across four challenging benchmarks—Charades-STA, QVHighlights, YouCookII, and Next-GQA—the method achieves state-of-the-art or competitive results, with clear improvements in temporal localization accuracy and descriptive quality, and strong zero-shot performance. The approach provides a practical, generalizable framework for temporally aware video tasks, with implications for robust video-centric AI systems and more reliable reasoning-grounded outputs.

Abstract

The core challenge in video understanding lies in perceiving dynamic content changes over time. However, multimodal large language models struggle with temporal-sensitive video tasks, which requires generating timestamps to mark the occurrence of specific events. Existing strategies require MLLMs to generate absolute or relative timestamps directly. We have observed that those MLLMs tend to rely more on language patterns than visual cues when generating timestamps, affecting their performance. To address this problem, we propose VideoExpert, a general-purpose MLLM suitable for several temporal-sensitive video tasks. Inspired by the expert concept, VideoExpert integrates two parallel modules: the Temporal Expert and the Spatial Expert. The Temporal Expert is responsible for modeling time sequences and performing temporal grounding. It processes high-frame-rate yet compressed tokens to capture dynamic variations in videos and includes a lightweight prediction head for precise event localization. The Spatial Expert focuses on content detail analysis and instruction following. It handles specially designed spatial tokens and language input, aiming to generate content-related responses. These two experts collaborate seamlessly via a special token, ensuring coordinated temporal grounding and content generation. Notably, the Temporal and Spatial Experts maintain independent parameter sets. By offloading temporal grounding from content generation, VideoExpert prevents text pattern biases in timestamp predictions. Moreover, we introduce a Spatial Compress module to obtain spatial tokens. This module filters and compresses patch tokens while preserving key information, delivering compact yet detail-rich input for the Spatial Expert. Extensive experiments demonstrate the effectiveness and versatility of the VideoExpert.

Figures (7)

• Figure 1: An example illustrating the temporal-sensitive video understanding tasks addressed by VideoExpert.
• Figure 2: The predicted result distributions of TimeChat timechat, VTimeLLM VTimeLLM and our VideoExpert on the Charades-STA twostage1 test split. Each point represents a predicted timestamp result. More prominent points indicate more frequent predictions. The contour plot shows the Ground-Truth distribution, where higher areas reflect more concentrated annotations. Both TimeChat and VTimeLLM frequently predict the same time range as result across different video-query pairs. This phenomenon becomes more pronounced when visual information is inaccurate or missing. These methods rely more on language patterns rather than visual cues when generating timestamps.
• Figure 3: Overview of the proposed VideoExpert for a series of temporal-sensitive video tasks.
• Figure 4: The pipeline of the Spatial Compress module.
• Figure 5: Effectiveness of Different Expert Ranks. The default LoRA rank in VideoExpert is fixed at $64$ to maintain a constant number of training parameters relative to previous works. The hyperparameter $\alpha$ controls the rank allocation between the two expert components. The rank of the temporal expert is defined as $64 \alpha$, while the spatial expert is given as $64(1 - \alpha)$.