See More, Store Less: Memory-Efficient Resolution for Video Moment Retrieval

TL;DR

SMORE addresses the memory bottlenecks in video moment retrieval by delivering dense, query-aligned representations under restricted hardware. It achieves this through three components: query-guided captioning to fuse user intent into captions, query-aware importance modulation to weigh informative content, and structured visual compression to remove redundancy via inter-frame similarity and rank-$k$ SVD. The approach yields state-of-the-art results on QVHighlights, Charades-STA, and ActivityNet-Captions while reducing memory usage, demonstrating practical feasibility for deployment on constrained GPUs. This work advances video-language understanding by showing that high retrieval accuracy and memory efficiency can co-occur, enabling broader accessibility and real-world applicability.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have improved image recognition and reasoning, but video-related tasks remain challenging due to memory constraints from dense frame processing. Existing Video Moment Retrieval (VMR) methodologies rely on sparse frame sampling, risking potential information loss, especially in lengthy videos. We propose SMORE (See MORE, store less), a framework that enhances memory efficiency while maintaining high information resolution. SMORE (1) uses query-guided captions to encode semantics aligned with user intent, (2) applies query-aware importance modulation to highlight relevant segments, and (3) adaptively compresses frames to preserve key content while reducing redundancy. This enables efficient video understanding without exceeding memory budgets. Experimental validation reveals that SMORE achieves state-of-the-art performance on QVHighlights, Charades-STA, and ActivityNet-Captions benchmarks.

Figures (6)

• Figure 1: Illustrates how MLLMs can be applied to VMR while considering a notion of information resolution: (a) Random or sparse sampling may miss key scenes due to low information resolution. (b) Zero-shot captioning offers dense coverage but lacks user intent, leading to less relevant captions for retrieval. (c) Our proposed SMORE yields dense and informative representation, preserving key details with query awareness.
• Figure 2: The overall architecture of SMORE. It first generates query-guided captions through QA (Sec.\ref{['subsec: Query-guided Captions for Retrieval']}). Next, query-aware importance modulation adjusts the relative importance between frames, captions, and queries (Sec.\ref{['subsec: Query-Aware Importance Modulation']}). By considering the information resolution, we efficiently reduce redundant information among the frame embeddings from the vision encoder (Sec.\ref{['subsec: Structured Frame Compression']}). The LLM encoder maps these frame tokens $f$ and caption tokens $c$ to their corresponding time embedding $t$ and interleaves them as input. Finally, the decoder outputs the temporal segment corresponding to the query.
• Figure 3: Query-guided caption generation. The query is parsed into objects and actions, which guide a QA-based relevance check. Relevant scenes receive query-aware prompts for captioning, improving alignment with retrieval goals.
• Figure 4: Illustration of the structured visual compression module. Redundant frame pairs are identified via cosine similarity and compressed using truncated SVD to produce compact, information-preserving embeddings.
• Figure 5: Qualitative results on the QVHighlights datasets.