LLM+MAP: Bimanual Robot Task Planning using Large Language Models and Planning Domain Definition Language

TL;DR

Problem: long-horizon bimanual manipulation requires robust coordination and reliable planning beyond what standalone LLMs typically provide. Approach: LLM+MAP grounds LLM reasoning in PDDL-based multi-agent planning, translating spatial scenes and natural-language tasks into executable representations and producing partial-order plans for two arms via symbolic solvers. Findings: in NICOL simulations across ServeWater, ServeFruit, and StackBlock domains, LLM+MAP achieves higher success rates and shorter planning times than direct LLM planning using GPT-4o, V3, o1, or R1, and benefits from multi-agent parallelism. Significance: this framework demonstrates a scalable integration of language-based reasoning with formal planning to improve robustness and efficiency in bimanual robotic task planning, with open-source code for replication.

Abstract

Bimanual robotic manipulation provides significant versatility, but also presents an inherent challenge due to the complexity involved in the spatial and temporal coordination between two hands. Existing works predominantly focus on attaining human-level manipulation skills for robotic hands, yet little attention has been paid to task planning on long-horizon timescales. With their outstanding in-context learning and zero-shot generation abilities, Large Language Models (LLMs) have been applied and grounded in diverse robotic embodiments to facilitate task planning. However, LLMs still suffer from errors in long-horizon reasoning and from hallucinations in complex robotic tasks, lacking a guarantee of logical correctness when generating the plan. Previous works, such as LLM+P, extended LLMs with symbolic planners. However, none have been successfully applied to bimanual robots. New challenges inevitably arise in bimanual manipulation, necessitating not only effective task decomposition but also efficient task allocation. To address these challenges, this paper introduces LLM+MAP, a bimanual planning framework that integrates LLM reasoning and multi-agent planning, automating effective and efficient bimanual task planning. We conduct simulated experiments on various long-horizon manipulation tasks of differing complexity. Our method is built using GPT-4o as the backend, and we compare its performance against plans generated directly by LLMs, including GPT-4o, V3 and also recent strong reasoning models o1 and R1. By analyzing metrics such as planning time, success rate, group debits, and planning-step reduction rate, we demonstrate the superior performance of LLM+MAP, while also providing insights into robotic reasoning. Code is available at https://github.com/Kchu/LLM-MAP.

Figures (5)

• Figure 1: Illustrative partial-order plan for bimanual manipulation (cf. Section \ref{['sec:method']}), where $C$, $A$ and $P$ indicate cup, area and point respectively. Actions for the left and right hands are colored in light blue and red respectively. Two boxes shown horizontally side by side represent two actions executed in parallel.
• Figure 2: Overview of our framework. According to the spatial description of the scene, with the bimanual domain knowledge and task description, LLM+MAP generates a PDDL representation that is used for multi-agent symbolic planning. Then, a valid partial-order plan is generated and executed by the NICOL bimanual robot (see Figure \ref{['fig:three_images']} for scenario setting).
• Figure 3: A visualization of the three task domains. In ServeWateer, the brown box is placed either in the left or right area to store the blue cup, while the cups and the human user are in random areas. In ServeFruit, the human stands exclusively in front of the overlap area to receive the bowl, while the fruits and the bowl are in random areas. In StackBlock, the blocks are distributed at random positions over the three areas, while the human user stands in front of a random area.
• Figure 4: The Group Debits statistics among successful tasks in three domains, the smaller the better.
• Figure 5: Planning Step Reduction Rate (%) of LLM+MAP over LLM+P, showcasing the improved efficiency of Multi-agent planning.