Robot Task Planning and Situation Handling in Open Worlds

TL;DR

The paper tackles open-world robotic task planning by introducing COWP, which augments a classical task planner with task-oriented commonsense extracted from an LLM to handle unforeseen situations. It combines a plan monitor and a knowledge acquirer to ground domain-specific action preconditions and effects, enabling dynamic replanning when faced with execution-time obstacles. Through a dining-domain dataset of 561 situations and a mobile manipulator demonstration, COWP outperforms closed-world, external-knowledge, and purely LM-based baselines. The work highlights the value of grounding LLM-derived commonsense in a structured planning framework and outlines directions for improved prompts, alternative LLMs, and richer perception and skills for real-world use.

Abstract

Automated task planning algorithms have been developed to help robots complete complex tasks that require multiple actions. Most of those algorithms have been developed for "closed worlds" assuming complete world knowledge is provided. However, the real world is generally open, and the robots frequently encounter unforeseen situations that can potentially break the planner's completeness. This paper introduces a novel algorithm (COWP) for open-world task planning and situation handling that dynamically augments the robot's action knowledge with task-oriented common sense. In particular, common sense is extracted from Large Language Models based on the current task at hand and robot skills. For systematic evaluations, we collected a dataset that includes 561 execution-time situations in a dining domain, where each situation corresponds to a state instance of a robot being potentially unable to complete a task using a solution that normally works. Experimental results show that our approach significantly outperforms competitive baselines from the literature in the success rate of service tasks. Additionally, we have demonstrated COWP using a mobile manipulator. The project website is available at: https://cowplanning.github.io/, where a more detailed version can also be found. This version has been accepted for publication in Autonomous Robots.

Figures (5)

• Figure 1: An illustrative example of a situation in the real world, encountered during the execution of the plan "delivering a cup to a human for drinking water." The robot approached a cabinet in a kitchen room on which a cup was located. The robot then found the cup to be delivered. Before grasping it, however, the robot detected a situation that the cup was occupied with a fork, a knife, and a spoon. This situation prevented the robot from performing the current action (i.e., grasping) and rendered normal solutions for drinking water invalid. A mobile service robot with a UR5e arm on a Segway RMP-110 base was used for demonstrations in this work.
• Figure 2: An overview of COWP that includes the three key components of Task Planner (provided as prior knowledge under closed-world assumption), Knowledge Acquirer, and Plan Monitor. The green (dashed) loop represents a plan execution process where the robot does encounter no situation, or these situations have no impact on the robot's plan execution. The orange loop is activated when the robot's current (closed-world) task planner is unable to develop a plan, which activates Knowledge Acquirer to augment the task planner with additional action effects utilizing common sense.
• Figure 3: Overall performances of COWP (ours) and three baseline methods under six different tasks, where the x-axis represents the task. Each success rate value is an average of 150 trials. The tasks are ranked based on the performance of COWP, where the very left corresponds to its best performance.
• Figure 4: Overall performances of COWP (ours) and three baseline methods under different objects, where the x-axis represents the object involved in the situation, the number beside each object is the occurrence of the object in our situation dataset, and the y-axis represents the success rate. The objects are ranked based on the performance of COWP, where the very left corresponds to its best performance.
• Figure 5: An illustrative example of COWP for open-world planning, where the robot was tasked with "delivering a cup for drinking water." (a) The robot walked to a cabinet, and located a cup on the cabinet. However, the robot found a situation that there were objects in the cup (a knife, a fork, and a spoon in this case). This observation was entered into the plan monitor, which queried GPT-3, and suggested that the planned action "grasp" was not applicable given the occupied cup. Accordingly, COWP updated its task planner by adding the new information that one cannot pour water into a non-empty cup. (b) The robot reasoned about other objects that were available in the environment, and queried GPT-3 to update the task planner about whether those objects can be used for drinking water -- details in Section \ref{['sec:alg']}. It happened that the robot learned a bowl could be used for drinking water. (c) A new plan of delivering a bowl to the human for drinking water was generated. Following the new plan, the robot walked to the table on which a bowl was located. (d) The robot grasped the bowl after observing it using vision. (e) The robot navigated to the dining table with the bowl. (f) The robot put down the bowl onto the dining table, and explained that a bowl was served due to the cup being occupied, which concluded the planning and execution processes.