SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM

TL;DR

SMART addresses video moment retrieval by combining audio-enhanced multimodal representations with shot-aware token compression to enable fine-grained temporal reasoning over long video sequences. The method uses a dual-branch encoder (visual via EVA-CLIP and Q-Former; audio via BEATs) and a LoRA-tuned LLM, forming a multimodal prompt that targets temporal segments. A two-stage STC module identifies keyframes and compresses within shots to reduce redundancy while preserving dynamic cues. Experiments on Charades-STA and QVHighlights show consistent state-of-the-art gains, especially for queries requiring precise temporal localization and cross-modal grounding, demonstrating improved efficiency and scalability. The work advances practical multimodal moment retrieval with potential impact on search, summarization, and surveillance applications.

Abstract

Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query. Despite recent progress in moment retrieval from videos using both traditional techniques and Multimodal Large Language Models (MLLM), most existing methods still rely on coarse temporal understanding and a single visual modality, limiting performance on complex videos. To address this, we introduce \textit{S}hot-aware \textit{M}ultimodal \textit{A}udio-enhanced \textit{R}etrieval of \textit{T}emporal \textit{S}egments (SMART), an MLLM-based framework that integrates audio cues and leverages shot-level temporal structure. SMART enriches multimodal representations by combining audio and visual features while applying \textbf{Shot-aware Token Compression}, which selectively retains high-information tokens within each shot to reduce redundancy and preserve fine-grained temporal details. We also refine prompt design to better utilize audio-visual cues. Evaluations on Charades-STA and QVHighlights show that SMART achieves significant improvements over state-of-the-art methods, including a 1.61\% increase in R1@0.5 and 2.59\% gain in R1@0.7 on Charades-STA.

Figures (6)

• Figure 1: Overview: (a) Traditional models rely on low-level visual features and generalize poorly. (b) MLLM-based models improve semantic understanding but ignore audio and incur high computational cost. (c) Our SMART model integrates audio and shot-coherent token compression for efficient, accurate, and generalizable multimodal moment retrieval.
• Figure 2: The Shot-aware Multimodal Audio-enhanced Retrieval of Temporal Segments (SMART) architecture:(a) The SMART pipeline integrates a pretrained MLLM with frozen visual and audio encoders and a lightweight LoRA-tuned LLM to predict temporal segments relevant to a query. Visual and audio features are projected, temporally encoded, and concatenated into a unified multimodal prompt, with shot-aware token compression applied for efficiency. (b) The Shot-aware Token Compression (STC, detailed in Fig. \ref{['fig:compression']}) operates in two stages: frames are first classified into key and non-key frames based on inter-frame differences; then, token-wise variance within each shot is analyzed. High-variance tokens (red), representing dynamic content, are retained, while low-variance tokens (blue) from non-key frames are discarded to remove redundancy while preserving essential cues.
• Figure 3: Shot-aware token compression: Key frames are detected via inter-frame differences. Within each shot, high-variance tokens from dynamic regions are preserved, while low-variance tokens from non-key frames are discarded, effectively minimizing redundancy and retaining essential temporal cues for accurate moment retrieval.
• Figure 4: Qualitative results on QVHighlights lei2021detecting: This figure visualizes the predicted and ground-truth segments for query events. SMART outperforms the visual-text MLLM baseline by preserving shot-level consistency and leveraging audio cues for fine-grained temporal understanding. See text for details.
• Figure 5: Impact of audio compression length $L$ of the Overall Concatenation strategy on QVHighlights, where the circular marker at $L=150$ indicates the optimal setting for all metrics.