MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generation

TL;DR

MaP-AVR addresses generalization gaps in robotic task planning by replacing human-centric skills with a universal meta-action set and grounding planning in Retrieval-Augmented Generation over a growing database of demonstrations. The method employs a fixed meta-action format {move/rotate, location description, end-effector status}, Chain-of-Thought prompts to generate plans, and RAG to retrieve similar demonstrations for in-context learning, with execution relying on VLM-based spatial reasoning to compute 6-DoF targets. Experiments in OmniGibson show substantial improvements over the state-of-the-art Rekep, especially when ICL is enabled, and ablations reveal the meta-action design and RAG as key drivers of performance. The work presents a scalable, integrable approach for embodied AI that can be extended to other robotic systems and tasks.

Abstract

Embodied robotic AI systems designed to manage complex daily tasks rely on a task planner to understand and decompose high-level tasks. While most research focuses on enhancing the task-understanding abilities of LLMs/VLMs through fine-tuning or chain-of-thought prompting, this paper argues that defining the planned skill set is equally crucial. To handle the complexity of daily environments, the skill set should possess a high degree of generalization ability. Empirically, more abstract expressions tend to be more generalizable. Therefore, we propose to abstract the planned result as a set of meta-actions. Each meta-action comprises three components: {move/rotate, end-effector status change, relationship with the environment}. This abstraction replaces human-centric concepts, such as grasping or pushing, with the robot's intrinsic functionalities. As a result, the planned outcomes align seamlessly with the complete range of actions that the robot is capable of performing. Furthermore, to ensure that the LLM/VLM accurately produces the desired meta-action format, we employ the Retrieval-Augmented Generation (RAG) technique, which leverages a database of human-annotated planning demonstrations to facilitate in-context learning. As the system successfully completes more tasks, the database will self-augment to continue supporting diversity. The meta-action set and its integration with RAG are two novel contributions of our planner, denoted as MaP-AVR, the meta-action planner for agents composed of VLM and RAG. To validate its efficacy, we design experiments using GPT-4o as the pre-trained LLM/VLM model and OmniGibson as our robotic platform. Our approach demonstrates promising performance compared to the current state-of-the-art method. Project page: https://map-avr.github.io/.

Figures (10)

• Figure 1: An example of a set of meta-actions as the planned outcome. Each meta-action comprises three essential components, which are utilized in subsequent execution functions.
• Figure 2: The figure illustrates that meta-actions represent the foundational abstraction for many skills aligned with human experience. The flexible combinations of meta-actions can be used to compose a wide range of daily tasks.
• Figure 3: This figure illustrates the interaction between the task database and the planner. The planner retrieves the most similar example for in-context learning, and successfully verified tasks are subsequently added to the database.
• Figure 4: The overview of our proposed MaP-AVR. Compared to previous methods, our pipeline differs in each of its major components. In understanding the task, we incorporate the Retrieval-Augmented Generation (RAG) technique to search the database for the closest successful task to facilitate in-context learning. In planning the task, we designed a series of Chain-of-Thought (CoT) prompts to guide the model generating meta-actions that align with expectations. In executing the task, we leverage the spatial understanding ability of VLMs, in conjunction with foundation models(such as Sam, DinoV2) and classical obstacle avoidance algorithms, to ensure that the planned meta-actions are executed effectively and successfully.
• Figure 5: This figure shows the linguistic structure of the meta-action we finally guide the VLM to generate.