Generative Recall, Dense Reranking: Learning Multi-View Semantic IDs for Efficient Text-to-Video Retrieval

TL;DR

This work tackles the scalability gap in text-to-video retrieval by introducing GRDR, a Generative Recall and Dense Reranking framework. It combines a multi-view video tokenizer that assigns multiple semantic IDs per video with a unified co-training scheme that shares a codebook between the tokenizer and a generative retriever, enabling end-to-end optimization and retrieval-aware semantic IDs. The recall stage performs trie-constrained, low-ambiguity ID generation, while a dense reranker preserves fine-grained semantics, yielding competitive accuracy with orders-of-magnitude storage reduction and up to 300x speedups on full-corpus retrieval. The approach effectively mitigates semantic ambiguity and cross-modal misalignment, making large-scale, real-time TVR more practical for industrial deployment.

Abstract

Text-to-Video Retrieval (TVR) is essential in video platforms. Dense retrieval with dual-modality encoders leads in accuracy, but its computation and storage scale poorly with corpus size. Thus, real-time large-scale applications adopt two-stage retrieval, where a fast recall model gathers a small candidate pool, which is reranked by an advanced dense retriever. Due to hugely reduced candidates, the reranking model can use any off-the-shelf dense retriever without hurting efficiency, meaning the recall model bounds two-stage TVR performance. Recently, generative retrieval (GR) replaces dense video embeddings with discrete semantic IDs and retrieves by decoding text queries into ID tokens. GR offers near-constant inference and storage complexity, and its semantic IDs capture high-level video features via quantization, making it ideal for quickly eliminating irrelevant candidates during recall. However, as a recall model in two-stage TVR, GR suffers from (i) semantic ambiguity, where each video satisfies diverse queries but is forced into one semantic ID; and (ii) cross-modal misalignment, as semantic IDs are solely derived from visual features without text supervision. We propose Generative Recall and Dense Reranking (GRDR), designing a novel GR method to uplift recalled candidate quality. GRDR assigns multiple semantic IDs to each video using a query-guided multi-view tokenizer exposing diverse semantic access paths, and jointly trains the tokenizer and generative retriever via a shared codebook to cast semantic IDs as the semantic bridge between texts and videos. At inference, trie-constrained decoding generates a compact candidate set reranked by a dense model for fine-grained matching. Experiments on TVR benchmarks show GRDR matches strong dense retrievers in accuracy while reducing index storage by an order of magnitude and accelerating up to 300$\times$ in full-corpus retrieval.

Figures (7)

• Figure 1: Effectiveness vs. Efficiency on MSRVTT-1k test set MSRVTT. Bubble size indicates the index storage required for cached video features. GRDR (Ours) achieves the optimal balance (positioned at the top-left).
• Figure 2: Overview of Text-to-Video Retrieval Paradigms. (a) Dense Retrieval: Decouples modalities into dual-encoders, relying on exhaustive similarity search over high-dimensional embeddings. (b) Generative Retrieval: Reformulates retrieval as sequence-to-sequence generation, constrained by a rigid one-to-one video-to-identifier mapping. (c) GRDR (Ours): leverage a Recall-then-Rerank paradigm using multi-view (one-to-many) tokenization to capture diverse video semantics.
• Figure 3: Overview of the GRDR framework. Training: (a) The multi-view video tokenizer encodes videos into discrete semantic IDs. It employs cross-modal alignment ($\mathcal{L}_{CL}$) to align video latent features $z_{i}$ with the retriever’s cumulative decoder features $h^{(m)}$ via contrastive learning. The video latent features $z_i$ then passed to residual quantization ($\mathcal{L}_{RQ}$), followed by a reconstruction decoder ($\mathcal{L}_{Rec}$) to prevent semantic collapse. (b) The generative retriever is jointly optimized with the tokenizer via a shared codebook. Its decoder predicts codes using cosine similarity, trained with cross-entropy loss ($\mathcal{L}_{CE}$). We employ progressive training, where Hierarchical Consistency ($\mathcal{L}_{HC}$) loss is used to prevent code drift in previous layers. The inference pipeline follows a Generative Recall, Dense Reranking paradigm: (a) Offline Indexing: Videos are tokenized into compact IDs to construct a prefix Trie for constrained decoding. (b) Online Retrieval: The retriever generates candidate Semantic IDs via beam search.
• Figure 4: Efficiency Scalability Analysis. (Left) Query-time latency $T_{latency}$ (ms) comparison between GRDR and dense retrieval across varying corpus sizes. (Right) Index storage requirements (MB) comparing GRDR's semantic IDs against dense frame-level and video-level features.
• Figure 5: Case study of retrieval results. T2VIndexer GT_TVR_T2VIndexer (Top) retrieves a video but only matching partial the subject. GRDR (Bottom) retrieves the ground-truth video containing the specific details from the query.