Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

R3G: A Reasoning--Retrieval--Reranking Framework for Vision-Centric Answer Generation

Zhuohong Chen, Zhengxian Wu, Zirui Liao, Shenao Jiang, Hangrui Xu, Yang Chen, Chaokui Su, Xiaoyu Liu, Haoqian Wang

TL;DR

This paper addresses vision-centric VQA by integrating external visual evidence through a two-stage retrieval pipeline and a reasoning-driven planning stage. The proposed R3G framework first generates a question-conditioned reasoning plan, then selects evidence images via coarse retrieval followed by a fine-grained, MLLM-based reranking that uses three sub-scores to assess semantic relevance, target correspondence, and answerability. By fusing coarse and fine cues and grounding answer generation in a planned reasoning path, R3G achieves state-of-the-art results on MRAG-Bench across multiple backbones, with ablations showing the complementary benefits of reasoning-before-evidence and the question-conditioned reranker. The approach highlights the importance of both choosing the right images and using them effectively, offering a reusable framework for vision-centric RAG in multimodal QA tasks.

Abstract

Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process. However, selecting the right images and integrating them effectively into the model's reasoning remains challenging.To address this challenge, we propose R3G, a modular Reasoning-Retrieval-Reranking framework.It first produces a brief reasoning plan that specifies the required visual cues, then adopts a two-stage strategy, with coarse retrieval followed by fine-grained reranking, to select evidence images.On MRAG-Bench, R3G improves accuracy across six MLLM backbones and nine sub-scenarios, achieving state-of-the-art overall performance. Ablations show that sufficiency-aware reranking and reasoning steps are complementary, helping the model both choose the right images and use them well. We release code and data at https://github.com/czh24/R3G.

R3G: A Reasoning--Retrieval--Reranking Framework for Vision-Centric Answer Generation

TL;DR

This paper addresses vision-centric VQA by integrating external visual evidence through a two-stage retrieval pipeline and a reasoning-driven planning stage. The proposed R3G framework first generates a question-conditioned reasoning plan, then selects evidence images via coarse retrieval followed by a fine-grained, MLLM-based reranking that uses three sub-scores to assess semantic relevance, target correspondence, and answerability. By fusing coarse and fine cues and grounding answer generation in a planned reasoning path, R3G achieves state-of-the-art results on MRAG-Bench across multiple backbones, with ablations showing the complementary benefits of reasoning-before-evidence and the question-conditioned reranker. The approach highlights the importance of both choosing the right images and using them effectively, offering a reusable framework for vision-centric RAG in multimodal QA tasks.

Abstract

Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process. However, selecting the right images and integrating them effectively into the model's reasoning remains challenging.To address this challenge, we propose R3G, a modular Reasoning-Retrieval-Reranking framework.It first produces a brief reasoning plan that specifies the required visual cues, then adopts a two-stage strategy, with coarse retrieval followed by fine-grained reranking, to select evidence images.On MRAG-Bench, R3G improves accuracy across six MLLM backbones and nine sub-scenarios, achieving state-of-the-art overall performance. Ablations show that sufficiency-aware reranking and reasoning steps are complementary, helping the model both choose the right images and use them well. We release code and data at https://github.com/czh24/R3G.
Paper Structure (12 sections, 9 equations, 1 figure, 4 tables)

This paper contains 12 sections, 9 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Reasoning-Before-Evidence
  4. Coarse Image Retrieval
  5. MLLM-as-Judge Reranking
  6. Experiments
  7. Dataset
  8. Implementation Details
  9. Main Results
  10. Ablation Studies
  11. Conclusion
  12. ACKNOWLEDGEMENTS

Figures (1)

  • Figure 1: Comparison of different methods for VQA.While MLLMs and text-based RAG often misalign with the question, vision-centric retrieval produces correct answers.