Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering

TL;DR

This work tackles KB-VQA efficiency by learning to compress retrieved knowledge into compact prompts that modulate a frozen MLLM via KV cache, reducing input token load without sacrificing accuracy. The method, RACC, comprises three phases: compression learning (with PIPE initialization and PRDB), information aggregation (DCSE and RGCA to enhance semantic content using image/question cues and retrieval scores), and modulation generation (P-Tuning v2 KV cache across MLLM layers). It achieves SOTA-like accuracy on OK-VQA (63.92% with RACC-hetero) and maintains strong performance on AOK-VQA, while delivering substantial latency and disk-space savings (22.0%–59.7% latency reduction; up to 91% on disk with pre-saved prompts). The approach is versatile, compatible with various MLLMs and knowledge sources (textual and multimodal), offering a practical path toward efficient, knowledge-rich VQA in real-world deployments.

Abstract

Multimodal large language models (MLLMs) have demonstrated great performance on visual question answering (VQA). When it comes to knowledge-based Visual Question Answering (KB-VQA), MLLMs may lack the specialized domain knowledge needed to answer questions, necessitating the retrieval of necessary information from external knowledge sources. Previous works like Retrival-Augmented VQA-v2 (RAVQA-v2) focus on utilizing as much input information, such as image-based textual descriptions and retrieved knowledge, as possible to improve performance, but they all overlook the issue that with the number of input tokens increasing, inference efficiency significantly decreases, which contradicts the demands of practical applications. To address this issue, we propose \textbf{R}etrieval-\textbf{A}ugmented MLLMs with Compressed Contexts (RACC). RACC learns to compress and aggregate retrieved knowledge for a given image-question pair, generating a compact modulation in the form of Key-Value (KV) cache to adapt the downstream frozen MLLM, thereby achieving effective and efficient inference. RACC achieves a state-of-the-art (SOTA) performance of 63.92\% on OK-VQA. Moreover, it significantly reduces inference latency by 22.0\%-59.7\% compared to the prominent RAVQA-v2. Abundant experiments show RACC's broad applicability. It is compatible with various off-the-shelf MLLMs and can also handle different knowledge sources including textual and multimodal documents.

Figures (2)

• Figure 1: The structural framework of RACC. An image and a question are first input into the multimodal retriever to retrieve $K$ relevant documents. During the compression learning phase, the $K$ documents, image, and question are input into HyperMLLM to obtain their corresponding compressed prompts. In the information aggregation phase, the obtained compressed prompts are aggregated to form the document-based compressed prompts of vision and question. In the modulation generation phase, the output of the aggregation network is processed by a set of MLPs to obtain the KV cache for each layer of the downstream BaseMLLM. At the same time, the BaseMLLM receives the image and question as input and generates the final answer.
• Figure 2: The VQA accuracy on the validation set during the training process of the framework using different methods.