Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Vision Meets Texts in Listwise Reranking

Hongyi Cai

TL;DR

Rank-Nexus tackles the problem of reranking image–text documents under a modality gap and data scarcity by introducing a lightweight 2B vision–language reranker. It employs a progressive cross-modal training curriculum and a diversity-driven data distillation pipeline to achieve strong text and multimodal reranking performance with minimal high-quality data. The approach delivers state-of-the-art results on MS MARCO text benchmarks (e.g., DL19/DL20) and competitive multimodal performance on INQUIRE and MMDocIR, while offering fast inference due to its compact size and non-reasoning architecture. The work highlights data quality and curriculum learning as critical levers for efficient multimodal IR, with practical implications for deploying multimodal reranking at scale.

Abstract

Recent advancements in information retrieval have highlighted the potential of integrating visual and textual information, yet effective reranking for image-text documents remains challenging due to the modality gap and scarcity of aligned datasets. Meanwhile, existing approaches often rely on large models (7B to 32B parameters) with reasoning-based distillation, incurring unnecessary computational overhead while primarily focusing on textual modalities. In this paper, we propose Rank-Nexus, a multimodal image-text document reranker that performs listwise qualitative reranking on retrieved lists incorporating both images and texts. To bridge the modality gap, we introduce a progressive cross-modal training strategy. We first train modalities separately: leveraging abundant text reranking data, we distill knowledge into the text branch. For images, where data is scarce, we construct distilled pairs from multimodal large language model (MLLM) captions on image retrieval benchmarks. Subsequently, we distill a joint image-text reranking dataset. Rank-Nexus achieves outstanding performance on text reranking benchmarks (TREC, BEIR) and the challenging image reranking benchmark (INQUIRE, MMDocIR), using only a lightweight 2B pretrained visual-language model. This efficient design ensures strong generalization across diverse multimodal scenarios without excessive parameters or reasoning overhead.

When Vision Meets Texts in Listwise Reranking

TL;DR

Rank-Nexus tackles the problem of reranking image–text documents under a modality gap and data scarcity by introducing a lightweight 2B vision–language reranker. It employs a progressive cross-modal training curriculum and a diversity-driven data distillation pipeline to achieve strong text and multimodal reranking performance with minimal high-quality data. The approach delivers state-of-the-art results on MS MARCO text benchmarks (e.g., DL19/DL20) and competitive multimodal performance on INQUIRE and MMDocIR, while offering fast inference due to its compact size and non-reasoning architecture. The work highlights data quality and curriculum learning as critical levers for efficient multimodal IR, with practical implications for deploying multimodal reranking at scale.

Abstract

Recent advancements in information retrieval have highlighted the potential of integrating visual and textual information, yet effective reranking for image-text documents remains challenging due to the modality gap and scarcity of aligned datasets. Meanwhile, existing approaches often rely on large models (7B to 32B parameters) with reasoning-based distillation, incurring unnecessary computational overhead while primarily focusing on textual modalities. In this paper, we propose Rank-Nexus, a multimodal image-text document reranker that performs listwise qualitative reranking on retrieved lists incorporating both images and texts. To bridge the modality gap, we introduce a progressive cross-modal training strategy. We first train modalities separately: leveraging abundant text reranking data, we distill knowledge into the text branch. For images, where data is scarce, we construct distilled pairs from multimodal large language model (MLLM) captions on image retrieval benchmarks. Subsequently, we distill a joint image-text reranking dataset. Rank-Nexus achieves outstanding performance on text reranking benchmarks (TREC, BEIR) and the challenging image reranking benchmark (INQUIRE, MMDocIR), using only a lightweight 2B pretrained visual-language model. This efficient design ensures strong generalization across diverse multimodal scenarios without excessive parameters or reasoning overhead.
Paper Structure (31 sections, 9 equations, 3 figures, 8 tables, 1 algorithm)

This paper contains 31 sections, 9 equations, 3 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Document Reranking
  4. Multimodal Retrieval
  5. Methodology
  6. Preliminary
  7. Problem Formulation
  8. Pairwise Reranking
  9. Listwise Ranking
  10. Vision-Language Embeddings
  11. Overview
  12. Text Reranking with Knowledge Distillation
  13. Data Quality over Quantity
  14. Image Reranking
  15. Quality-Aware Filtering.
  16. ...and 16 more sections

Figures (3)

  • Figure 1: The effectivenss (left) and latency (right) of Rank-Nexus and other baseline method on document listwise reranking inference. We experimented and illustrated DL-19 benchmark for evaluation and one of distilled MSMARCO dataset as inference time example. Rank-R1 takes most latency among 3 models, strongly dependent on reasoning time. While Rank-Nexus are the fastest as it possesses smallest parameter size and outputs without ad-hoc thinking.
  • Figure 2: Overview of Rank-Nexus training pipeline. Stage-1: We distill listwise rankings from Claude-4.5 on diverse multimodal documents, generating structured output with relevance ordering and explanations. Stage-2: Quality filtering removes low-confidence samples (CLIP score threshold) while diversity-based coreset selection ensures coverage across semantic clusters, producing high-quality training data. Stage-3: Progressive curriculum trains the model sequentially on text listwise ranking, image pairwise ranking, and image listwise ranking to build comprehensive multimodal reranking capabilities.
  • Figure 3: Diminishing return on data scaling. We have experimented on 1k, 4k, 8k and 10k of MSMARCO training dataset for fine-tuning in Rank-Nexus-2B-Qwen.