Enrich and Detect: Video Temporal Grounding with Multimodal LLMs

TL;DR

This work tackles fine-grained video temporal grounding by marrying multimodal LLMs with a cascaded enrich-and-detect pipeline. The model first enriches user queries with missing contextual details derived from the video, then uses a lightweight interval decoder to predict precise temporal boundaries, conditioned on enriched representations. Training employs an MIL strategy to choose between the original and enriched queries, mitigating hallucinations and improving grounding reliability. Empirically, ED-VTG achieves state-of-the-art results across zero-shot STG and VPG benchmarks and remains competitive with specialist models in fine-tuned settings, highlighting the practicality and generalization of LLM-based grounding for complex video understanding.

Abstract

We introduce ED-VTG, a method for fine-grained video temporal grounding utilizing multi-modal large language models. Our approach harnesses the capabilities of multimodal LLMs to jointly process text and video, in order to effectively localize natural language queries in videos through a two-stage process. Rather than being directly grounded, language queries are initially transformed into enriched sentences that incorporate missing details and cues to aid in grounding. In the second stage, these enriched queries are grounded, using a lightweight decoder, which specializes at predicting accurate boundaries conditioned on contextualized representations of the enriched queries. To mitigate noise and reduce the impact of hallucinations, our model is trained with a multiple-instance-learning objective that dynamically selects the optimal version of the query for each training sample. We demonstrate state-of-the-art results across various benchmarks in temporal video grounding and paragraph grounding settings. Experiments reveal that our method significantly outperforms all previously proposed LLM-based temporal grounding approaches and is either superior or comparable to specialized models, while maintaining a clear advantage against them in zero-shot evaluation scenarios.

Figures (8)

• Figure 1: Our proposed system. ED-VTG performs video temporal grounding as a two-stage process: since user queries are often incomplete or coarse, the first stage involves producing an enriched query that adds additional details to the original, making it easier to ground. Meanwhile, a contextualized embedding is generated, containing all the information about the interval to be predicted. In the second stage, an interval decoder translates these embeddings into precise temporal boundaries.
• Figure 2: Overview of the proposed Enrich and Detect framework: (left) ED-VTG pipeline: Given an untrimmed video and a query $Q$ to be grounded, the inputs are first tokenized into video tokens $T_V$ and text tokens $T_Q$. The tokens are then fed into an LLM, which first generates an enriched query by, e.g., filling in any missing details and then emitting an interval token $\texttt{<INT>}$. The embedding of this special token is finally decoded into the predicted temporal interval via a lightweight interval decoder. In the example shown here, the vague input query is enriched into a more detailed one by our model which can be subsequently grounded more easily. (right) Training: ED-VTG is trained using ground-truth temporal intervals and pseudo-labels of enriched queries, generated by an external off-the-shelf captioning model li2024llavaonevision, which -- unlike our model at inference time -- has access to the ground-truth intervals. For every sample during training, the proposed multiple instance learning (MIL) framework allows ED-VTG to assess both the original or the enriched queries and generate two sets of predictions, $\hat{I}^{enr}$, the interval predicted using the enriched query, and $\hat{I}^{dir}$ using the original query. Next, the model backpropagates using the better prediction (i.e., lower grounding loss). Hence, during training, ED-VTG dynamically learns to decide for which sample enrichment is necessary and, based on that, performs detection.
• Figure 3: Example of query enrichment and detection made by ED-VTG on video paragraph grounding (VPG) task from the ActivityNet-Captions krishna2017dense dataset. In this specific sample, three different queries are enriched and localized together.
• Figure A.1: Examples of query enrichment and localization made by ED-VTG on single-query temporal grounding (STG) task from the Charades-STA gao2017charadessta dataset. We also show the prediction made by one baseline model, TimeChat ren2024timechat, which directly ground the input queries using raw-text timestamp representation. Since we train ED-VTG using the MIL paradigm, the model can choose to use the input query directly or enrich it during evaluation. In the last example, since the input query is clear and explicit, the model directly localizes it.
• Figure A.2: Comparison of detections of ED-VTG using its predicted enriched queries against a baseline version trained to always use the original queries. The enriched queries contain additional relevant details and context that enable ED-VTG to perform more accurate temporal localization. In the first example, which is taken from Charades-STA gao2017charadessta, the additional details in the enriched query provide a more complete description of objects and actions that is more easily groundable. In the second example, sourced from the ActivityNet-Captions krishna2017dense dataset, the enriched query provides additional temporal context which leads to more precise temporal boundary prediction.