Anticipate, Adapt, Act: A Hybrid Framework for Task Planning

TL;DR

This work presents a hybrid framework that integrates the generic prediction capabilities of an LLM with the probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language and demonstrates substantial improvement in performance compared with state of the art baselines.

Abstract

Anticipating and adapting to failures is a key capability robots need to collaborate effectively with humans in complex domains. This continues to be a challenge despite the impressive performance of state of the art AI planning systems and Large Language Models (LLMs) because of the uncertainty associated with the tasks and their outcomes. Toward addressing this challenge, we present a hybrid framework that integrates the generic prediction capabilities of an LLM with the probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language. For any given task, the robot reasons about the task and the capabilities of the human attempting to complete it; predicts potential failures due to lack of ability (in the human) or lack of relevant domain objects; and executes actions to prevent such failures or recover from them. Experimental evaluation in the VirtualHome 3D simulation environment demonstrates substantial improvement in performance compared with state of the art baselines.

Figures (5)

• Figure 1: Illustrative task of fetching a glass of water from the sink to the kitchen counter. In the baseline scenario, the human may end up dropping the water glass due to stability issues. Our framework enables the robot to anticipate such failures; it either prevents the failure by completing the task, or prepares to recover from the failure by fetching a mop that can be used to clean the potential water spill.
• Figure 2: Our framework's pipeline: (a) LLM takes a prompt of task lists, user preferences, scene description, and current command, to predict upcoming tasks; (b) RDDL description of domain knowledge and joint goals comprising current and predicted tasks fed to PROST planner; and (c) Plan of actions to be executed by robot and human to achieve the goal, including robot's actions to prevent or recover from potential failures due to the human's actions.
• Figure 3: Partial description of reward specification for the $PrepareBreakfast(Toast)$ task.
• Figure 4: Example probabilities of state transition probabilities in human behavior model. The probability of the human: walking to the kitchen from a random location is 0.8; grabbing a bread slice while in the kitchen is 0.625; placing the bread slice in the toaster after grabbing it is 0.6; switching on the toaster while in the kitchen is 0.625.
• Figure 5: Task failures as a function of the reward-based multiplier that determines extent to which failure prevention and recovery is prioritized over task completion. With an increase in the multiplier, failures reduce up to a point before increasing again. A trade off between failure prevention and task completion leads to good performance.