Reason to Contrast: A Cascaded Multimodal Retrieval Framework

TL;DR

This work extends TTE-v2, a hybrid multimodal retrieval framework that introduces reasoning-driven performance scaling based on additional input token budget rather than model or embedding size, and highlights the promise of token-wise scaling as an alternative scaling paradigm for multimodal retrieval.

Abstract

Traditional multimodal retrieval systems rely primarily on bi-encoder architectures, where performance is closely tied to embedding dimensionality. Recent work, Think-Then-Embed (TTE), shows that incorporating multimodal reasoning to elicit additional informative tokens before embedding can further improve retrieval. In this paper, we extend this paradigm with TTE-v2, a hybrid multimodal retrieval framework that introduces reasoning-driven performance scaling based on additional input token budget rather than model or embedding size. Our approach augments the initial multimodal retrieval with additional reasoning steps for reranking, enabling more expressive query-candidate interactions at test time. The reranking stage further provides fine-grained supervision for hard negative mining and false negative filtering, creating a feedback loop that effectively strengthens the upstream retriever. This cascaded design delivers substantial test-time improvements based on intermediate reasoning token scaling. Experiments on the MMEB-V2 benchmark demonstrate that TTE-v2-7B achieves a new state-of-the-art accuracy of 75.7%, and that TTE-v2-2B matches or surpasses leading 7B models trained with significantly larger external data. Our results highlight the promise of token-wise scaling as an alternative scaling paradigm for multimodal retrieval.

Figures (15)

• Figure 1: TTE-v2 improves test-time scaling of multimodal retrieval. We compare performance $w.r.t.$ budget of averaged number of tokens per query. Model sizes are reflected in the marker sizes. We consider 2B and 7B embedder sizes. MM Ranker refers to using MLLM (Qwen2.5 VL qwen2.5vl 7B, 32B and 72B) as a computational-expensive baseline, to directly rank all the candidates in the retrieval pool. $\mathrm{TTE}\text{-v2}$$\mathrm{ECRR}$ refers to our method that reranks the candidates retrieved by the embedder. $\mathrm{TTE}\text{-v2}$$\mathrm{ECRR}$ + $\mathrm{QAR}$ adds an additional query-target joint reasoning step to introduce more discriminative information for reranking.
• Figure 2: Comparison between different embedding and retrieval frameworks. Top left: traditional embed-then-retrieve. Top right: recently proposed think-then-embed ($\mathrm{TTE}$) tte procedure. Bottom: the reason-to-contrast framework ($\mathrm{TTE}\text{-v2}$) proposed in this work.
• Figure 3: Overall pipeline of our Reason-to-Contrast framework ($\mathrm{TTE}\text{-v2}$). In stage 1, we follow the TTE framework: given a query, we retrieve top-$n$ target videos and their ECRs. In Stage 2, we perform the proposed Reason-to-Contrast framework, which consists of Query-Aware Reasoning ($\mathrm{QAR}$) stage on the retrieved candidates to incorporate more discriminative information, followed by a ECR-based reranking stage ($\mathrm{ECRR}$).
• Figure 4: Comparison between the original ECR of a video and the ECR after Query-Aware Rewriting (QAR), from the VATEX T2V test set.
• Figure 5: Comparison between top-$1$ and top-$10$ on retrieval tasks in MMEB V2 for TTE-2B (top) and 7B (bottom). For image and video, we show precision, and NDCG for visdoc. We separately show video retrieval tasks (colored in dark green), while other tasks are averaged by task types.