Universal Video Temporal Grounding with Generative Multi-modal Large Language Models

TL;DR

The paper tackles universal video temporal grounding by introducing UniTime, a framework that interleaves timestamp tokens with multi-scale video tokens and leverages generative Multi-modal LLMs to localize natural language queries in videos of varying length and domain. It combines adaptive frame scaling, timestamp-based alignment, and coarse-to-fine multi-stage inference, along with a video-centric training paradigm to enable efficient, scalable grounding. Empirical results show UniTime achieves state-of-the-art or competitive performance across short- and long-video VTG benchmarks, excels in zero-shot settings, and enhances downstream long-form VideoQA. The approach demonstrates strong generalization, flexibility across base MLLMs, and practical value for real-world, heterogeneous video understanding tasks.

Abstract

This paper presents a computational model for universal video temporal grounding, which accurately localizes temporal moments in videos based on natural language queries (e.g., questions or descriptions). Unlike existing methods that are often limited to specific video domains or durations, we propose UniTime, a robust and universal video grounding model leveraging the strong vision-language understanding capabilities of generative Multi-modal Large Language Models (MLLMs). Our model effectively handles videos of diverse views, genres, and lengths while comprehending complex language queries. The key contributions include: (i) We consider steering strong MLLMs for temporal grounding in videos. To enable precise timestamp outputs, we incorporate temporal information by interleaving timestamp tokens with video tokens. (ii) By training the model to handle videos with different input granularities through adaptive frame scaling, our approach achieves robust temporal grounding for both short and long videos. (iii) Comprehensive experiments show that UniTime outperforms state-of-the-art approaches in both zero-shot and dataset-specific finetuned settings across five public temporal grounding benchmarks. (iv) When employed as a preliminary moment retriever for long-form video question-answering (VideoQA), UniTime significantly improves VideoQA accuracy, highlighting its value for complex video understanding tasks.

Figures (11)

• Figure 1: The UniTime framework empowers MLLMs with advanced universal temporal grounding capabilities. (a) UniTime can handle diverse videos with various views, genres, and durations, as well as comprehend complex language queries. (b) UniTime achieves universal temporal grounding through a coarse-to-fine approach. (c) Performance comparison on temporal grounding and video question answering benchmarks demonstrates the superior capabilities of UniTime.
• Figure 2: Overview of the proposed UniTime framework. (a) UniTime achieves universal temporal grounding by leveraging adaptive frame scaling to construct multi-scale video inputs and then generate multi-scale predictions, allowing robust grounding across diverse video durations. (b) Within the model architecture, UniTime constructs an interleaved sequence of timestamps and scaled frame features, which, combined with the language query, is fed into the LLM and then identifies the corresponding temporal interval from the timestamp tokens.
• Figure 3: Ablation on hyperparameters. We decouple the effects of Segment Length (left) and Replication Factor (right) on temporal grounding performance.
• Figure 4: Temporal distribution bias in Charades-STA.Darker colors indicate higher data density.
• Figure 6: Illustration of the video-centric training paradigm.