GranAlign: Granularity-Aware Alignment Framework for Zero-Shot Video Moment Retrieval

TL;DR

GranAlign tackles granularity mismatch in zero-shot video moment retrieval by introducing a training-free, dual-granularity alignment framework. It reformulates queries into simplified and detailed variants and generates query-agnostic and query-aware captions, computing a granular moment score that blends broad recall with fine-grained precision. The score combines two aligned pairs, ($Q_s$, $C_{agn}$) and ($Q_d$, $C_{awr}$), via $S_f = \frac{1}{2m} \sum_{i=1}^{m} \left[ \operatorname{g}(q_{s}^{(i)}, C_{agn,f}) + \operatorname{g}(q_{d}^{(i)}, C_{awr,f}) \right]$, followed by length-regularized span scoring and NMS to predict the final moment. Empirically, GranAlign achieves state-of-the-art results on QVHighlights, Charades-STA, and ActivityNet-Captions without task-specific training, demonstrating robust cross-dataset generalization and effective handling of diverse query granularities.

Abstract

Zero-shot video moment retrieval (ZVMR) is the task of localizing a temporal moment within an untrimmed video using a natural language query without relying on task-specific training data. The primary challenge in this setting lies in the mismatch in semantic granularity between textual queries and visual content. Previous studies in ZVMR have attempted to achieve alignment by leveraging high-quality pre-trained knowledge that represents video and language in a joint space. However, these approaches failed to balance the semantic granularity between the pre-trained knowledge provided by each modality for a given scene. As a result, despite the high quality of each modality's representations, the mismatch in granularity led to inaccurate retrieval. In this paper, we propose a training-free framework, called Granularity-Aware Alignment (GranAlign), that bridges this gap between coarse and fine semantic representations. Our approach introduces two complementary techniques: granularity-based query rewriting to generate varied semantic granularities, and query-aware caption generation to embed query intent into video content. By pairing multi-level queries with both query-agnostic and query-aware captions, we effectively resolve semantic mismatches. As a result, our method sets a new state-of-the-art across all three major benchmarks (QVHighlights, Charades-STA, ActivityNet-Captions), with a notable 3.23% mAP@avg improvement on the challenging QVHighlights dataset.

Figures (10)

• Figure 1: An illustration of how GranAlign resolves the 'Granularity Mismatch'. The (a) and (b) fail to localize all moments due to a mismatch in semantic granularity, resulting in low precision and low coverage, respectively. In contrast, our proposed GranAlign (c) integrates both granular levels, achieving both high precision and high coverage to correctly localize all target moments. Takeaway: GranAlign overcomes the core 'Granularity Mismatch' by synergizing the high-recall simple path with the high-precision detailed path.
• Figure 2: Comparison of our method (b) against a baseline (a) (without considering the granularity of query and video content) on the QVHighlights validation set, categorized by query type (see Section 4.3 for categorization criteria).
• Figure 3: Conceptual Framework Comparison. (a) Previous methods typically adopt a single-path approach, reformulating the query. (b) Our method, GranAlign, employs a dual-path framework, decomposing the query into Simplified and Detailed versions and aligning them with Query-Agnostic and Query-Aware captions, respectively.
• Figure 4: Overview of the GranAlign Framework. In Granularity-Aware Alignment (Sec.\ref{['sec: scoring']}), the input query is rewritten at two semantic granularities (simplified/detailed) and matched with either query-agnostic or query-aware captions (captions generated with the query as context) to obtain a Moment Score for each video segment. These scores drive the Moment Proposal Generator and NMS stage, after which the post-ranking module produces the final prediction (Sec.\ref{['sec: proposal']}). Takeaway: GranAlign effectively tackles the ZVMR task by leveraging fine-grained semantic alignment between queries and video content.
• Figure 5: Examples of instruction pairs for granularity-based query rewriting. This figure illustrates sample prompts given to LLaMA-3 to generate simplified (recall-focused) and detailed (precision-focused) queries.