MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding

TL;DR

MUSEG addresses the gap in fine-grained temporal reasoning for multimodal LLMs by introducing timestamp-aware multi-segment grounding trained with a phased RL recipe. It defines a segment matching reward $r_M$ and a timestamp reward $r_T$, enabling temporally grounded reasoning over multiple video segments. The method demonstrates substantial improvements on temporal grounding benchmarks and time-sensitive video QA, with analyses validating the importance of multi-segment grounding and phased rewards. This approach advances practical temporal understanding in video-language models and suggests directions for broader temporal reasoning capabilities.

Abstract

Video temporal understanding is crucial for multimodal large language models (MLLMs) to reason over events in videos. Despite recent advances in general video understanding, current MLLMs still struggle with fine-grained temporal reasoning. While reinforcement learning (RL) has been explored to address this issue recently, existing RL approaches remain limited in effectiveness. In this work, we propose MUSEG, a novel RL-based method that enhances temporal understanding by introducing timestamp-aware multi-segment grounding. MUSEG enables MLLMs to align queries with multiple relevant video segments, promoting more comprehensive temporal reasoning. To facilitate effective learning, we design a customized RL training recipe with phased rewards that progressively guides the model toward temporally grounded reasoning. Extensive experiments on temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG significantly outperforms existing methods and generalizes well across diverse temporal understanding scenarios. View our project at https://github.com/THUNLP-MT/MUSEG.

Figures (7)

• Figure 1: Performance of MUSEG-7B on various temporal grounding (Charades-STA, THUMOS14 and THUMOS15) and broader time-sensitive video understanding (E.T. Bench Subset) tasks.
• Figure 2: An example comparing our MUSEG-7B with previous models. MUSEG-7B performs more precise, timestamp-aware reasoning by leveraging multiple key temporal cues to derive the correct answer.
• Figure 3: Overview of MUSEG. (a) Our proposed segment matching reward (up) and timestamp reward (down). (b) RL-based training process with phased rewards of MUSEG.
• Figure 4: Cases of MUSEG-7B on multi-segment grounding (in domain) and referred action recognition (out of domain) tasks.
• Figure 5: Segment matching reward (a) w/o local matching, (b) w/ local matching (sequential), and (c) w/ local matching (maximum). (d) Evolution of numbers of predicted segments during training process. For all the plots, we only consider queries whose groundtruths are more than one segments.