RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models

TL;DR

RoboMP$^2$ addresses the generalization limitations of multimodal large language models in embodied robotics by decoupling perception and planning into GCMP and RAMP. GCMP uses a tailored multimodal model to semantically interpret complex referential expressions and locate targets, while RAMP employs a coarse-to-fine retrieval strategy to bring the most relevant demonstrations into prompts for multimodal plan generation with GPT4V. The approach yields about a 10% improvement on the VIMA benchmark and up to ~40% gains in real-world tasks, demonstrating stronger generalization to unseen scenarios and robustness to multimodal context. This framework significantly enhances autonomous robotic manipulation by integrating rich multimodal environment information into planning, enabling more reliable sim-to-real transfer and scalable operation in diverse settings.

Abstract

Multimodal Large Language Models (MLLMs) have shown impressive reasoning abilities and general intelligence in various domains. It inspires researchers to train end-to-end MLLMs or utilize large models to generate policies with human-selected prompts for embodied agents. However, these methods exhibit limited generalization capabilities on unseen tasks or scenarios, and overlook the multimodal environment information which is critical for robots to make decisions. In this paper, we introduce a novel Robotic Multimodal Perception-Planning (RoboMP$^2$) framework for robotic manipulation which consists of a Goal-Conditioned Multimodal Preceptor (GCMP) and a Retrieval-Augmented Multimodal Planner (RAMP). Specially, GCMP captures environment states by employing a tailored MLLMs for embodied agents with the abilities of semantic reasoning and localization. RAMP utilizes coarse-to-fine retrieval method to find the $k$ most-relevant policies as in-context demonstrations to enhance the planner. Extensive experiments demonstrate the superiority of RoboMP$^2$ on both VIMA benchmark and real-world tasks, with around 10% improvement over the baselines.

Figures (8)

• Figure 1: (a) The detection results of the yellow block with the complex spatial reference using different methods; (b) Different plans for different environment, even if the same instruction.
• Figure 1: The results on the VIMA Benchmark. $^\dag$ denotes the cited result.
• Figure 2: Overview of our proposed RoboMP$^2$ framework. The three parts in grey/blue/green represent the input data, planning and perception, respectively. The modules highlighted are trainable, including fusion module and LoRA.
• Figure 3: Examples of manipulated objects with four different referential relationships types.
• Figure 4: An example of training data for the GCMP.