Anticipate & Act : Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments

TL;DR

This work addresses the inefficiency of executing household tasks sequentially by integrating LLM-driven high-level task anticipation with a classical planning system (PDDL/Fast Downward) to produce joint, fine-grained action sequences. By prompting LLMs with routine examples, the agent anticipates upcoming tasks and grounds them as joint goals in a domain model, enabling the planner to interleave actions across tasks. Experimental results in VirtualHome show a notable 31% reduction in execution time and a 12% reduction in plan length when anticipating multiple tasks, with GPT-4 achieving robust task-order predictions given contextual prompts. The framework demonstrates practical benefits for assistive agents in dynamic home environments and points to future work on scalability, probabilistic planning, and real-robot deployment.

Abstract

Assistive agents performing household tasks such as making the bed or cooking breakfast often compute and execute actions that accomplish one task at a time. However, efficiency can be improved by anticipating upcoming tasks and computing an action sequence that jointly achieves these tasks. State-of-the-art methods for task anticipation use data-driven deep networks and Large Language Models (LLMs), but they do so at the level of high-level tasks and/or require many training examples. Our framework leverages the generic knowledge of LLMs through a small number of prompts to perform high-level task anticipation, using the anticipated tasks as goals in a classical planning system to compute a sequence of finer-granularity actions that jointly achieve these goals. We ground and evaluate our framework's abilities in realistic scenarios in the VirtualHome environment and demonstrate a 31% reduction in execution time compared with a system that does not consider upcoming tasks.

Figures (7)

• Figure 1: Anticipation example: (a) Agent individually moves the milk and then the food to the desk; (b) Agent anticipates that milk needs to be served after food, jointly moving them to eliminate an extra trip.
• Figure 2: Our framework's pipeline: (a) user inputs prompts with sequences of household tasks to an LLM, which then predicts high-level tasks over a time horizon; (b) the sequence of tasks is mapped to a joint goal state in a finer-granularity domain description in an action language (PDDL); (c) a heuristic planner (FD) uses this description to jointly compute the sequence of actions to be executed to complete all the tasks; and (d) the plan is executed in a realistic simulation environment. $^*$Actions corresponding to fold clothes are omitted due to space restrictions.
• Figure 3: Action for Dusting the object $o$ at some location $l$.
• Figure 4: Example plans produced with and without considering anticipated tasks.
• Figure 5: LLM prompting example.