Number it: Temporal Grounding Videos like Flipping Manga

TL;DR

The paper tackles the challenge of precise temporal grounding in Video Large Language Models by introducing Number-Prompt (NumPro), which overlays readable frame numbers onto video frames to transform VTG into a visual-timeline alignment task without architectural changes. It further extends to NumPro-FT, a fine-tuning variant that uses a NumPro-augmented dataset and LoRA-based parameter-efficient training to achieve state-of-the-art VTG on Charades-STA and ActivityNet, across moment retrieval and highlight detection. Through CLIP-guided design, attention analysis, and extensive experiments across multiple Vid-LLMs and VTG benchmarks, NumPro demonstrates strong, transfer-ready gains with minimal impact on general video QA. The work highlights NumPro’s robustness, scalability, and potential for broad adoption in end-to-end video-dialog systems, while also discussing limitations and future directions for longer videos and adaptive frame-rate strategies.

Abstract

Video Large Language Models (Vid-LLMs) have made remarkable advancements in comprehending video content for QA dialogue. However, they struggle to extend this visual understanding to tasks requiring precise temporal localization, known as Video Temporal Grounding (VTG). To address this gap, we introduce Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual comprehension with temporal grounding by adding unique numerical identifiers to each video frame. Treating a video as a sequence of numbered frame images, NumPro transforms VTG into an intuitive process: flipping through manga panels in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking visual content with corresponding temporal information. Our experiments demonstrate that NumPro significantly boosts VTG performance of top-tier Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and 8.5\% in mAP for highlight detection. The code will be available at https://github.com/yongliang-wu/NumPro.

Figures (13)

• Figure 1: Effectiveness of Adding Frame Numbers for Temporal Grounding: (a) Without numbered images or frames, both humans and Vid-LLMs struggle to locate specific timestamps accurately. (b) Once numbered, grounding temporal cues becomes as intuitive as flipping manga, where timestamps are accessible at a glance.
• Figure 2: Attention Analysis between Video Frames and Event Query. Although the model accurately attends to regions of interest related to the query, it struggles to generate precise temporal boundaries in its response.
• Figure 3: Framework of Our Approach in Two Settings: (1) Training-free VTG with NumPro, where frame numbers are directly added to video frames, enabling Vid-LLMs to locate events temporally without additional training, and (2) Fine-tuned VTG with NumPro-FT, which further improves VTG performance by fine-tuning Vid-LLMs on a dataset NumPro-enhanced with no architectural modifications.
• Figure 4: Our NumPro Design Algorithm. We overlay different numbers onto COCO images and obtain visual and textual representations using CLIP encoders. For each configuration, we calculate Number/Caption Similarity and derive Number/Caption Accuracy, to identify the optimal NumPro design that balances recognizability and minimal disruption to the visual content.
• Figure 5: The Impact of Different Number-Prompt Designs. We categorize the design into three dimensions: font size, position, and color. BL stands for Bottom Left, BR for Bottom Right, TL for Top Left, TR for Top Right, and C for Center.