CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planning

TL;DR

The work introduces Multimodal Retrieval Augmented Generation Planning (MRAG Planning) to address the inefficiencies of traditional MRAG pipelines by optimizing information acquisition and query reformulation. It presents CogPlanner, a plug-and-play planning framework with parallel and sequential modeling, and CogBench, a dedicated benchmark with large-scale data to evaluate planning decisions and enable lightweight integration with resource-efficient MLLMs. Empirical results show CogPlanner improves end-to-end MRAG performance across multiple backbones, achieving substantial gains with minimal additional cost, and demonstrate the viability of deploying a lightweight planning expert (e.g., Qwen2-7B-VL-Cog) for real-world applications. The work highlights the importance of adaptive planning for multi-hop multimodal queries and provides a path toward more robust, efficient MRAG systems that can scale to diverse domains.

Abstract

Multimodal Retrieval Augmented Generation (MRAG) systems have shown promise in enhancing the generation capabilities of multimodal large language models (MLLMs). However, existing MRAG frameworks primarily adhere to rigid, single-step retrieval strategies that fail to address real-world challenges of information acquisition and query reformulation. In this work, we introduce the task of Multimodal Retrieval Augmented Generation Planning (MRAG Planning) that aims at effective information seeking and integration while minimizing computational overhead. Specifically, we propose CogPlanner, an agentic plug-and-play framework inspired by human cognitive processes, which iteratively determines query reformulation and retrieval strategies to generate accurate and contextually relevant responses. CogPlanner supports parallel and sequential modeling paradigms. Furthermore, we introduce CogBench, a new benchmark designed to rigorously evaluate the MRAG Planning task and facilitate lightweight CogPlanner integration with resource-efficient MLLMs, such as Qwen2-VL-7B-Cog. Experimental results demonstrate that CogPlanner significantly outperforms existing MRAG baselines, offering improvements in both accuracy and efficiency with minimal additional computational costs.

Figures (4)

• Figure 1: An example of current MRAG system with the benefits of incorporating MRAG Planning.
• Figure 2: The overall framework of CogPlanner.
• Figure 3: Domain distribution of CogBench.
• Figure 4: The average number of retrieval actions of different MLLMs across queries with different reasoning steps.