Towards Zero-Knowledge Task Planning via a Language-based Approach

TL;DR

The paper defines the Zero-Knowledge Task Planning problem where an agent must achieve a goal without task-specific information, relying on a fixed action set and real-time sensory input. It proposes a language-based pipeline centered on a Natural Language Planning Engine that decomposes instructions into subgoals and generates Behavior Trees, plus a Refinement Loop that patches plans during execution based on feedback, demonstrated in AI2-THOR. The authors formalize ZKTP, implement an end-to-end system using an LLM with prompts to interpret, decompose, plan, and refine, and compare against several baselines that rely on task-specific data. Key findings show competitive or superior task success rates in complex, unseen tasks with zero task knowledge, albeit with higher planning overhead, underscoring the potential of language-driven planning for zero-shot robotic tasks. This work lays groundwork for scalable, adaptable autonomous agents that can operate in dynamic environments without prior task-specific setup, highlighting practical impact in robotics and intelligent automation.

Abstract

In this work, we introduce and formalize the Zero-Knowledge Task Planning (ZKTP) problem, i.e., formulating a sequence of actions to achieve some goal without task-specific knowledge. Additionally, we present a first investigation and approach for ZKTP that leverages a large language model (LLM) to decompose natural language instructions into subtasks and generate behavior trees (BTs) for execution. If errors arise during task execution, the approach also uses an LLM to adjust the BTs on-the-fly in a refinement loop. Experimental validation in the AI2-THOR simulator demonstrate our approach's effectiveness in improving overall task performance compared to alternative approaches that leverage task-specific knowledge. Our work demonstrates the potential of LLMs to effectively address several aspects of the ZKTP problem, providing a robust framework for automated behavior generation with no task-specific setup.

Figures (2)

• Figure 1: We present a language-based approach that addresses what we call the Zero-Knowledge Task Planning (ZKTP) problem, i.e., planning high-level actions to achieve a given goal without any task-specific information. Our approach takes as input a natural language instruction, interprets a task goal from sensory input (e.g., images of the environment), breaks down the goal into a hierarchy of more manageable subtasks, generates Behavior Trees for completing a given task, then patches these Behavior Trees as needed if errors occur during task execution. The components of our approach interface via automatically generated prompts that incorporate information from a previous component then receive a text output from a large language model (LLM).
• Figure 2: In the task shown, the robot in the AI2-Thor environment is given the instruction "set a place at the dining table". Using Zero-Knowledge task planning, the robot here reasons about the environment and figures out on-the-fly that this instruction must mean placing a plate, fork, and knife on the table, and the robot correctly generates actions in order to successfully accomplish these sub-goals. The left image here is the initial state of the environment, the middle image shows the robot finding and picking up the fork, and the right image shows the final state achieved at the end of task execution.