Anticipate & Collab: Data-driven Task Anticipation and Knowledge-driven Planning for Human-robot Collaboration

TL;DR

DaTAPlan addresses task anticipation for human–robot collaboration by coupling LLM-based high-level task predictions with a classical planner that generates a joint agent–human action sequence. The framework maps anticipated tasks into a PDDL problem and uses the Fast Downward planner to minimize $C(\pi)=\sum_{m} c_m^{\mathcal{R}}+\sum_{n} c_n^{\mathcal{H}}$, yielding adaptively executable plans $\pi^* = \arg \min_{\pi} C(\pi)$. It also supports automatic replanning when human outcomes or preferences deviate, and re-prompts the LLM to reflect new goals. Experiments in CoppeliaSim with diverse households demonstrate improved planning efficiency, reduced execution time, effective collaboration, and robust adaptation, including collision avoidance. The work offers a practical path toward transparent, data-informed yet knowledge-grounded assistive robotics in real-world domestic settings.

Abstract

An agent assisting humans in daily living activities can collaborate more effectively by anticipating upcoming tasks. Data-driven methods represent the state of the art in task anticipation, planning, and related problems, but these methods are resource-hungry and opaque. Our prior work introduced a proof of concept framework that used an LLM to anticipate 3 high-level tasks that served as goals for a classical planning system that computed a sequence of low-level actions for the agent to achieve these goals. This paper describes DaTAPlan, our framework that significantly extends our prior work toward human-robot collaboration. Specifically, DaTAPlan planner computes actions for an agent and a human to collaboratively and jointly achieve the tasks anticipated by the LLM, and the agent automatically adapts to unexpected changes in human action outcomes and preferences. We evaluate DaTAPlan capabilities in a realistic simulation environment, demonstrating accurate task anticipation, effective human-robot collaboration, and the ability to adapt to unexpected changes. Project website: https://dataplan-hrc.github.io

Figures (7)

• Figure 1: Illustration of "human-robot collaboration with anticipation": (a) agent anticipates (serving task) and collaborates with human, fetching juice from the fridge to the table while fetching the egg that the human cooks in a metal pot and brings to the table; (b) The agent only serves the juice to the table and the human entirely performs the necessary actions needed to cook and serve the egg.
• Figure 2: Our framework's pipeline: (a) Input prompt contains the list of possible tasks, user preferences, and scene description, along with an example prompt and the corresponding output high-level tasks; (b) High-level tasks predicted by LLM are mapped to PDDL problem description; (c) The FD planner generates a plan of agent's actions and the expected human actions; (d) Deviations of the human from the expected plan are noted and used to trigger replanning when appropriate.
• Figure 3: Few shot prompting with LLMs.
• Figure 4: Action for boiling an item ?o. Precondition: item must be in the metal pot.
• Figure 5: Evaluating H2. Values of execution cost and plan length with different levels of anticipation computed as a ratio over values computed for no anticipation; paired trials conducted for different scenarios in two different households. The combination of task anticipation and action planning improved performance.