TA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchors

TL;DR

This work tackles the challenge of temporally grounding dense video captions by introducing TA-Prompting, which leverages Temporal Anchors to localize events and steer VideoLLMs for temporally aware captioning. A two-stage training framework combines an event localizer (predicting $M$ anchors) with a temporal-aware captioning module, while an inference-time Event Coherent Sampling (ECS) enforces cross-event coherence and faithful grounding via a combined score $\mathcal S=\mathcal S_{cs}+\alpha\mathcal S_{as}$. Empirical results on ActivityNet Captions, YouCook2, Charades-STA, and ActivityNet-RTL show improvements in dense video captioning (CIDEr, SODA_c, METEOR), event localization (Recall/Precision/F1), moment retrieval (mIoU, R@IoU), and TemporalQA, demonstrating stronger temporal grounding and narrative coherence. The approach also includes ablations and qualitative analyses that highlight the importance of temporal anchors and ECS for robust, temporally consistent descriptions, suggesting practical impact for long-video understanding and cross-modal reasoning tasks.

Abstract

Dense video captioning aims to interpret and describe all temporally localized events throughout an input video. Recent state-of-the-art methods leverage large language models (LLMs) to provide detailed moment descriptions for video data. However, existing VideoLLMs remain challenging in identifying precise event boundaries in untrimmed videos, causing the generated captions to be not properly grounded. In this paper, we propose TA-Prompting, which enhances VideoLLMs via Temporal Anchors that learn to precisely localize events and prompt the VideoLLMs to perform temporal-aware video event understanding. During inference, in order to properly determine the output caption sequence from an arbitrary number of events presented within a video, we introduce an event coherent sampling strategy to select event captions with sufficient coherence across temporal events and cross-modal similarity with the given video. Through extensive experiments on benchmark datasets, we show that our TA-Prompting is favorable against state-of-the-art VideoLLMs, yielding superior performance on dense video captioning and temporal understanding tasks including moment retrieval and temporalQA.

Figures (10)

• Figure 1: Comparisons between standard VideoLLMs and TA-Prompting. Instead of using text tokens to describe time information, our TA-Prompting employs temporal anchors to precisely localize events, which steers VideoLLM to caption the grounded video segment.
• Figure 2: Overview of TA-Prompting. (A) TA-Prompting learns to predict temporal anchors $\hat{Y}$ for each event in a video with an event localizer $\theta_{\mathcal{E}}$. (B) With temporal anchors as proposals, TA-Prompting performs precise event localization and predicts temporal-aware event captions. Note that a time embedding layer $\theta_{\mathcal{T}}$ and a LoRA are jointly trained with $\theta_{\mathcal{E}}$ in (B).
• Figure 3: Evaluation of dense video captioning on the ActivityNet Captions datasets. The models to be compared with are LITA (blue), VTG-LLM (red), VTimeLLM (yellow), and TA-Prompting (Ours) (green). Note that the metrics of CIDEr and METEOR are calculated across different IoU thresholds of ${0.3, 0.5, 0.7}$.
• Figure 4: Qualitative evaluation and comparisons of predicted timestamps and the output captions. The width of each caption block represents the event duration, denoting the corresponding starting and ending time. Note that captions in green indicate correct alignment with the ground truth ones, while the captions in red indicate descriptions that are irrelevant to the visual content.
• Figure 5: Visualization of a failure case with qualitative comparison of predicted timestamps and captions. The captions in green indicate correct alignment with the ground truth ones, while the captions in red indicate descriptions that are irrelevant to the visual content.