Efficient Multimodal Planning Agent for Visual Question-Answering

TL;DR

The paper tackles inefficiencies in rigid multimodal RAG pipelines for visual question answering by introducing a multimodal planning agent that dynamically selects necessary mRAG steps per query. It formalizes the problem with notations $q=(\mathbf{i},\mathbf{t})$, contexts $\mathbf{k_i},\mathbf{k_t}$, and a gold query $\mathbf{q}_g$, and trains the agent to predict one of four retrieval modes using a prompts-based objective $J(\theta)$. Through a data-annotation pipeline including Visual Query Decomposition to produce $q_i$, $a_i$, and $q_g$, the agent learns to omit redundant steps and intelligently apply image/text retrieval as needed, achieving substantial efficiency gains. Experiments across six VQA datasets show the method reduces search time by over 60% on average vs OmniSearch and yields lower tool-call latency (3x faster than WebWatcher 7B and 4.5x faster than WebWatcher 32B) while improving or maintaining task performance. Overall, the work demonstrates that adaptive, planning-based multimodal reasoning can markedly improve scalability and responsiveness in knowledge-intensive VQA tasks, with strong cross-model applicability using LoRA-fine-tuned agents.

Abstract

Visual Question-Answering (VQA) is a challenging multimodal task that requires integrating visual and textual information to generate accurate responses. While multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a multimodal planning agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the agent to intelligently determine the necessity of each mRAG step. In our experiments, the agent can help reduce redundant computations, cutting search time by over 60\% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research agent and a carefully designed prompt-based method, on average over six various datasets. Code will be released.

Figures (5)

• Figure 1: Workflow of our agent on solving VQA with dynamic mRAG strategies. The agent selects a sub-path based on different VQA inputs, which may require image search, query search, neither, or both.
• Figure 2: Proposed data annotation method.
• Figure 3: Comparison to OmniSearch with performance and detailed retrieval counts. The layered bars show the search counts of each i2i, t2t, and t2i type.
• Figure 4: Training and evaluation dynamics.
• Figure 5: Three failure cases.