Question-Focused Filtering for Knowledge-based VQA

TL;DR

This work tackles knowledge-based visual question answering by addressing two core filtering errors: article selection and intra-article information retrieval. It introduces QKVQA, a two-stage filtering framework consisting of a trainable Question-Focused Filter (QFF) and a Chunk-based Dynamic Multi-Article Selection (CDA), designed to fuse question semantics with cross-article knowledge while maintaining efficiency. QFF uses a Q-Former to inject question guidance into section representations, and CDA performs fine-grained, chunk-level filtering across multiple articles with dynamic quotas, aided by a reranker for chunk relevance. Experiments on E-VQA and InfoSeek demonstrate consistent, substantial improvements over state-of-the-art baselines, confirming the approach effectively balances deep semantic understanding with practical computational costs and enabling robust cross-article knowledge integration.

Abstract

Knowledge-based Visual Question Answering (KB-VQA) aims to answer questions by integrating images with external knowledge. Effective knowledge filtering is crucial for improving accuracy. Typical filtering methods use similarity metrics to locate relevant article sections from one article, leading to information selection errors at the article and intra-article levels. Although recent explorations of Multimodal Large Language Model (MLLM)-based filtering methods demonstrate superior semantic understanding and cross-article filtering capabilities, their high computational cost limits practical application. To address these issues, this paper proposes a question-focused filtering method. This approach can perform question-focused, cross-article filtering, efficiently obtaining high-quality filtered knowledge while keeping computational costs comparable to typical methods. Specifically, we design a trainable Question-Focused Filter (QFF) and a Chunk-based Dynamic Multi-Article Selection (CDA) module, which collectively alleviate information selection errors at both the article and intra-article levels. Experiments show that our method outperforms current state-of-the-art models by 4.9% on E-VQA and 3.8% on InfoSeek, validating its effectiveness. The code is publicly available at: https://github.com/leaffeall/QKVQA.

Figures (6)

• Figure 1: Comparison between typical filtering methods and our proposed approach.
• Figure 2: Concrete examples of the two types of errors.
• Figure 3: The figure illustrates our complete pipeline, in which the question‑focused filtering framework consists of components (2) and (3).
• Figure 4: Qualitative comparison between OMGM and our proposed QKVQA method on E-VQA (top row) and InfoSeek (bottom row) image-question pairs. For OMGM in each row: the first two cases show article selection errors, while the last one exhibits intra-article information selection errors. Our QKVQA method successfully alleviates these issues and generates more accurate answers.
• Figure 5: Additional qualitative results on image-question pairs from Encyclopedic-VQA, where we compare the answers provided by QKVQA with those from OMGM. The first two rows demonstrate article selection errors by OMGM, while the last row shows an intra-article information selection error. Our QKVQA method provides correct answers in all cases.