ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Models

TL;DR

ET-Plan-Bench introduces an automatic embodied planning benchmark and evaluation pipeline to assess foundation models' spatial and temporal cognition in interactive environments. It leverages multi-source simulators (e.g., Virtual Home, Habitat) to provide immediate feedback enabling dynamic replanning by LLM agents. The benchmark defines navigation and manipulation tasks with spatial constraints (relations, occlusions, layouts) and temporal dependencies, supported by an automated task-generation and ground-truth labeling pipeline. Experiments show that state-of-the-art models struggle as tasks grow in spatial-temporal complexity, but small models can match larger ones with supervised fine-tuning, illustrating scalable pathways to improve embodied planning through data and prompts.

Abstract

Recent advancements in Large Language Models (LLMs) have spurred numerous attempts to apply these technologies to embodied tasks, particularly focusing on high-level task planning and task decomposition. To further explore this area, we introduce a new embodied task planning benchmark, ET-Plan-Bench, which specifically targets embodied task planning using LLMs. It features a controllable and diverse set of embodied tasks varying in different levels of difficulties and complexities, and is designed to evaluate two critical dimensions of LLMs' application in embodied task understanding: spatial (relation constraint, occlusion for target objects) and temporal & causal understanding of the sequence of actions in the environment. By using multi-source simulators as the backend simulator, it can provide immediate environment feedback to LLMs, which enables LLMs to interact dynamically with the environment and re-plan as necessary. We evaluated the state-of-the-art open source and closed source foundation models, including GPT-4, LLAMA and Mistral on our proposed benchmark. While they perform adequately well on simple navigation tasks, their performance can significantly deteriorate when faced with tasks that require a deeper understanding of spatial, temporal, and causal relationships. Thus, our benchmark distinguishes itself as a large-scale, quantifiable, highly automated, and fine-grained diagnostic framework that presents a significant challenge to the latest foundation models. We hope it can spark and drive further research in embodied task planning using foundation models.

Figures (26)

• Figure 1: Evaluation task statistics of ET-Plan-Bench, which includes a diverse set of navigation and manipulation tasks, along with their advanced versions with spatial and temporal constraints.
• Figure 2: Task generation pipeline. Task-specific requirements and scene graph from the simulator are used as inputs for automatic task generation. More details are given in Section \ref{['sec:TaskGenerationPileline']}
• Figure 3: The LLM agent pipeline for evaluation comprises an automatic prompt selection module, a navigation module, and a manipulation module. More details are given in Section \ref{['sec:eval_pipeline']}
• Figure 4: An example of a spatial constrained task. The robot explores different rooms and wall shelves to find one in which the relationship constraint matches the goal of the task. The images were generated using the Virtual Home simulator.
• Figure 5: In a mapless task scenario, the robot begins in the bedroom, gradually explores the living room and kitchen, and ultimately discovers the bowl. The images were generated using the Habitat 2.0 simulator.