VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding

TL;DR

<VTG-LLM> addresses the critical need for precise timestamp localization in VTG by integrating timestamp knowledge into visual tokens through sequence-time embedding, absolute-time tokens, and slot-based token compression. It introduces VTG-IT-120K, a high-quality, multi-task VTG instruction-tuning dataset, to train and evaluate the model. Empirical results show strong zero-shot VTG performance across Youcook2, Charades-STA, and QVHighlights, with ablations confirming the benefits of time embeddings, formatting of time tokens, and the slot-based compression. The work advances practical VTG capabilities for video LLMs and offers a high-quality dataset resource for future research, alongside limitations such as focus on timestamp accuracy and absence of audio modalities.

Abstract

Video Temporal Grounding (VTG) strives to accurately pinpoint event timestamps in a specific video using linguistic queries, significantly impacting downstream tasks like video browsing and editing. Unlike traditional task-specific models, Video Large Language Models (video LLMs) can handle multiple tasks concurrently in a zero-shot manner. Consequently, exploring the application of video LLMs for VTG tasks has become a burgeoning research area. However, despite considerable advancements in video content understanding, video LLMs often struggle to accurately pinpoint timestamps within videos, limiting their effectiveness in VTG tasks. To address this, we introduce VTG-LLM, a model designed to enhance video LLMs' timestamp localization abilities. Our approach includes: (1) effectively integrating timestamp knowledge into visual tokens; (2) incorporating absolute-time tokens to manage timestamp knowledge without concept shifts; and (3) introducing a lightweight, high-performance, slot-based token compression technique designed to accommodate the demands of a large number of frames to be sampled for VTG tasks. Additionally, we present VTG-IT-120K, a collection of publicly available VTG datasets that we have re-annotated to improve upon low-quality annotations. Our comprehensive experiments demonstrate the superior performance of VTG-LLM in comparison to other video LLM methods across a variety of VTG tasks.

Figures (8)

• Figure 1: Overview of the VTG-LLM model.
• Figure 2: Overview of slot based token compression.
• Figure 3: Example of new annotations.
• Figure 4: Quality analysis.
• Figure 5: Annotation example of moment retrieval task.