Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering

TL;DR

This work tackles knowledge-based visual question answering (KB-VQA) by requiring robust integration of external knowledge with visual content. It introduces Wiki-PRF, a Processing–Retrieval–Filtering (PRF) framework that enables dynamic visual-tool processing to generate precise multimodal queries, multimodal retrieval over a large knowledge base, and a filtering stage that condenses retrieval results for accurate answering. A visual language model, VLM-PRF, is trained via reinforcement learning with an answer-accuracy reward and a format-consistency reward, using LoRA for efficient adaptation; the reward is defined as $r_{\1}(x,y) = \alpha EM(a_{pred},a_{gt}) + \beta M(a_{tool},t_{tool}) + \gamma M(a_{filter},t_{filter})$, with $\alpha=2.0$, $\beta=1.0$, and $\gamma=1.0$. Experiments on E-VQA and InfoSeek establish state-of-the-art results (e.g., 36.0 on E-VQA and 42.8 on InfoSeek for Wiki-PRF with RL), and ablations demonstrate the contributions of processing, retrieval, and filtering stages, the value of tool-based retrieval, and the robustness to knowledge-base scale. The work highlights the potential of RL-guided, tool-augmented multimodal RAG for KB-VQA and provides code for reproducibility.

Abstract

Knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-base querying, it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retrieval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, we introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning, tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality, achieving state-of-the-art performance. Code is available at https://github.com/cqu-student/Wiki-PRF

Figures (12)

• Figure 2: Overview of Wiki-PRF. Wiki-PRF comprises three key stages: (1) Processing Stage: VLM-PRF processes the input image and its corresponding question using external tools. (2) Retrieval Stage: Relevant Wikipedia articles are retrieved, split into individual sections, and ranked based on their similarity to the processed input. (3) Filtering Stage: The re-ranked article sections are further filtered by VLM-PRF to retain the most relevant content, which is then fed into the VLM for final answer generation. During training, VLM-PRF is supervised using two types of reward signals: answer reward , which evaluates the correctness of the generated answer, and format reward , which ensures the output adheres to the desired structure.
• Figure 3: Example of the tool calls and filtering. By analyzing the problem, VLM-PRF performs captioning, grounding and flipping operations on the images. Using the retrieved sections, VLM-PRF performs filtering and generates the task-oriented results.
• Figure 4: Qualitaive examples of Wiki-PRF.
• Figure 5: The training curve of VLM-PRF-7B in E-VQA.
• Figure 6: Comparison on E-VQA