MMRAG-RFT: Two-stage Reinforcement Fine-tuning for Explainable Multi-modal Retrieval-augmented Generation

TL;DR

Addresses the explainability gap in multi-modal retrieval-augmented generation by introducing a two-stage reinforcement fine-tuning framework that first filters candidates with coarse point-wise ranking and then optimizes fine-grained list-wise ranking and answer generation with explainable prompts. The approach enables transparent reasoning in both retrieval and generation, validated on WebQA, MultimodalQA, and the Mini-WebQA dataset, achieving state-of-the-art results and robust ablations. Key contributions include the rule-based RL for initial filtering, reasoning-based RL for detailed ranking and QA, and explicit explainable chain-of-thought mechanisms. This work advances practical MMRAG by delivering credible, interpretable outputs with data-efficient training.

Abstract

Multi-modal Retrieval-Augmented Generation (MMRAG) enables highly credible generation by integrating external multi-modal knowledge, thus demonstrating impressive performance in complex multi-modal scenarios. However, existing MMRAG methods fail to clarify the reasoning logic behind retrieval and response generation, which limits the explainability of the results. To address this gap, we propose to introduce reinforcement learning into multi-modal retrieval-augmented generation, enhancing the reasoning capabilities of multi-modal large language models through a two-stage reinforcement fine-tuning framework to achieve explainable multi-modal retrieval-augmented generation. Specifically, in the first stage, rule-based reinforcement fine-tuning is employed to perform coarse-grained point-wise ranking of multi-modal documents, effectively filtering out those that are significantly irrelevant. In the second stage, reasoning-based reinforcement fine-tuning is utilized to jointly optimize fine-grained list-wise ranking and answer generation, guiding multi-modal large language models to output explainable reasoning logic in the MMRAG process. Our method achieves state-of-the-art results on WebQA and MultimodalQA, two benchmark datasets for multi-modal retrieval-augmented generation, and its effectiveness is validated through comprehensive ablation experiments.

Figures (2)

• Figure 1: An example of explainable multi-modal retrieval-augmented generation. The reinforcement fine-tuned model obtains the most relevant document and the final answer based on clear reasoning logic.
• Figure 2: Illustration of our proposed two-stage reinforcement fine-tuning framework.