RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval

TL;DR

RankVideo tackles the bottleneck of text-to-video retrieval by introducing a video-native reasoning reranker that operates directly on audiovisual inputs. It employs a two-stage curriculum—perception-grounded supervised fine-tuning to learn grounded video captions, followed by ranking fine-tuning with a joint objective that includes pointwise calibration, pairwise ranking, and teacher distillation—augmented by a data synthesis pipeline to generate reasoning-intensive queries. On the large MultiVENT 2.0 benchmark, RankVideo delivers substantial gains, averaging a 31% improvement in nDCG@10 over various first-stage retrievers and surpassing text-only and vision-language rerankers, while maintaining efficiency. The approach demonstrates strong generalization across first-stage models and supports downstream benefits in retrieval-augmented generation, highlighting its practical impact for scalable, high-quality video retrieval.

Abstract

Reranking is a critical component of modern retrieval systems, which typically pair an efficient first-stage retriever with a more expressive model to refine results. While large reasoning models have driven rapid progress in text-centric reranking, reasoning-based reranking for video retrieval remains underexplored. To address this gap, we introduce RANKVIDEO, a reasoning-based reranker for video retrieval that explicitly reasons over query-video pairs using video content to assess relevance. RANKVIDEO is trained using a two-stage curriculum consisting of perception-grounded supervised fine-tuning followed by reranking training that combines pointwise, pairwise, and teacher confidence distillation objectives, and is supported by a data synthesis pipeline for constructing reasoning-intensive query-video pairs. Experiments on the large-scale MultiVENT 2.0 benchmark demonstrate that RANKVIDEO consistently improves retrieval performance within a two-stage framework, yielding an average improvement of 31% on nDCG@10 and outperforming text-only and vision-language reranking alternatives, while more efficient.

Figures (14)

• Figure 1: RankVideo judges the relevance between a query-video pair, dynamically reasoning or answering depending on the difficulty of the query-video pair.
• Figure 2: RankVideo is trained with a two-stage process. Stage 1 uses a perception-grounded supervised finetuning, where the model learns to generate captions grounded in video content. In Stage 2, for each text query, we sample a query grouped batch containing one positive (relevant) video and one or more negatives (not relevant), and score each candidate using the difference between the logits for yes and no. The model is optimized with a combined objective: (1) teacher-probability distillation toward $p_{yes}$, (2) a pointwise loss for stable binary calibration, and (3) pairwise ranking loss that pushes the positive to the top within the query batch.
• Figure 3: Training Stage-2 increases score separation in the reranking regime. Empirical CDF of the reranker score $s_{\theta}(q,v) = \ell_\theta(\textbf{yes}|q,v) - \ell_\theta(\textbf{no}|q,v)$ for relevant and non relevant query video pairs. Stage 2 shifts relevant pairs towards larger positive margins and suppresses non relevant candidates towards more negative margins, reducing overlaps in the score distributions within reranking candidate pools.
• Figure 4: Median query latency for Qwen3VL Instruct/Thinking (QVL-I/T) and RankVideo stages 1 and 2. Latency is computed as the mean for 100 query-video pairs with a batch size of 1. All evaluations are run with batch size 1 as larger batches exceed GPU memory for VLMs.
• Figure 5: RankVideo nDCG@10 by query event type. Multi-word categories are abbreviated as acronyms in the plot (e.g., PD=Political Development, LD=Launch/Discovery, SE=Social Events). Only attributes with $\ge$ 30 test queries are included.