Context-Enhanced Video Moment Retrieval with Large Language Models

TL;DR

The paper addresses the challenging problem of localizing moments in untrimmed videos given complex, free-form language queries. It proposes LLM-guided Moment Retrieval (LMR), which combines LLM-derived target-related context with visual features via Video Context Modeling and a DETR-inspired language-conditioned transformer to decode queries and predict moments. Key contributions include the LLM-based Video Context Enhancement, Video Context Modeling with Video-Query Fusion, and a Complex Query Validation (C-QVal) setup with strong empirical gains on QVHighlights and Charades-STA, especially for complex queries. The approach demonstrates that incorporating rich, LLM-generated contextual information significantly improves semantic alignment and localization accuracy, offering practical benefits for video analysis tasks requiring nuanced contextual understanding.

Abstract

Current methods for Video Moment Retrieval (VMR) struggle to align complex situations involving specific environmental details, character descriptions, and action narratives. To tackle this issue, we propose a Large Language Model-guided Moment Retrieval (LMR) approach that employs the extensive knowledge of Large Language Models (LLMs) to improve video context representation as well as cross-modal alignment, facilitating accurate localization of target moments. Specifically, LMR introduces a context enhancement technique with LLMs to generate crucial target-related context semantics. These semantics are integrated with visual features for producing discriminative video representations. Finally, a language-conditioned transformer is designed to decode free-form language queries, on the fly, using aligned video representations for moment retrieval. Extensive experiments demonstrate that LMR achieves state-of-the-art results, outperforming the nearest competitor by up to 3.28\% and 4.06\% on the challenging QVHighlights and Charades-STA benchmarks, respectively. More importantly, the performance gains are significantly higher for localization of complex queries.

Figures (5)

• Figure 1: (a) We introduce LLM-based context enhancement and modeling to improve video representations and directly employ free-form language as queries to decode these representations for accurate moment localization. (b) The benchmark QD-DETR moon2023query predicts false positives due to insufficient context modeling and decoding (captures major context but misses important details), whereas, our approach accurately localizes the target moment.
• Figure 2: Architecture of LMR. Given a video sequence and a query language, LLM-based Video Context Enhancement generates target-related context information, Video Context Modeling enhances video representations using target-related visual features and LLM-based video context, and Language-conditioned Transformer directly decodes free-form language queries using aligned video representations for moment retrieval.
• Figure 3: Improvements over the state-of-the-art (QD-DETR moon2023query) on mAP and R1 metrics on the Charades dataset.
• Figure 4: Qualitative results. (a)(b): queries with specific backgrounds, (c)(d): queries in specific contextual events, (e): queries with particular appearances, (f): queries specifying certain actions. The above constraints are highlighted in yellow in the queries. The yellow rectangular boxes represent moments semantically aligning with the query without yellow-highlighted scene constraints. We observed that QD-DETR moon2023query tends to localize the moments missing the background and action details.
• Figure 5: Comparison of attention values with/without specific information.