Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Look Further Ahead: Testing the Limits of GPT-4 in Path Planning

Mohamed Aghzal, Erion Plaku, Ziyu Yao

TL;DR

It is found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4’s path planning effectiveness, but these approaches do not obtain optimal paths and fail at generalizing over extended horizons.

Abstract

Large Language Models (LLMs) have shown impressive capabilities across a wide variety of tasks. However, they still face challenges with long-horizon planning. To study this, we propose path planning tasks as a platform to evaluate LLMs' ability to navigate long trajectories under geometric constraints. Our proposed benchmark systematically tests path-planning skills in complex settings. Using this, we examined GPT-4's planning abilities using various task representations and prompting approaches. We found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4's path planning effectiveness. However, while these approaches show some promise toward improving the planning ability of the model, they do not obtain optimal paths and fail at generalizing over extended horizons.

Look Further Ahead: Testing the Limits of GPT-4 in Path Planning

TL;DR

It is found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4’s path planning effectiveness, but these approaches do not obtain optimal paths and fail at generalizing over extended horizons.

Abstract

Large Language Models (LLMs) have shown impressive capabilities across a wide variety of tasks. However, they still face challenges with long-horizon planning. To study this, we propose path planning tasks as a platform to evaluate LLMs' ability to navigate long trajectories under geometric constraints. Our proposed benchmark systematically tests path-planning skills in complex settings. Using this, we examined GPT-4's planning abilities using various task representations and prompting approaches. We found that framing prompts as Python code and decomposing long trajectory tasks improve GPT-4's path planning effectiveness. However, while these approaches show some promise toward improving the planning ability of the model, they do not obtain optimal paths and fail at generalizing over extended horizons.
Paper Structure (21 sections, 4 figures, 3 tables)

This paper contains 21 sections, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. LLMs as Autonomous Planning Agents
  4. Benchmarks for LLM Path Planning
  5. MATERIALS AND METHODS
  6. Benchmark Data Synthesis
  7. Geometric Environments
  8. Ground-Truth Plan Generation
  9. Task Representations
  10. Prompting Methodologies
  11. Model and Implementation
  12. Evaluation
  13. RESULTS AND ANALYSIS
  14. Planning with Different Task Representations
  15. Planning in Different Geometries
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Summary of the different environment types used in the experiments. The black regions represent the obstacles (walls), while the white space represents free cells. The figure shows one instance from each of the three types: (a) rectangular blocks, where certain regions are completely blocked, (b) square mazes, alternating squares with a single opening on each square, and (c) zig-zag mazes, consisting of horizontal obstacles on alternating rows except one opening.
  • Figure 2: Overview of our planning with feedback prompting method using the different representations. The example shown is of a rectangular blocks setting. Solutions are highlighted in purple, initial locations are highlighted in blue, while goal locations are highlighted in green. Environmental feedback consists of a warning message (highlighted in pink) explaining the cause behind failure, and the current status of the agent after performing the actions (highlighted in orange).
  • Figure 3: Path planning performance ($y$-axis) achieved using different prompt methodologies as a function of the ground-truth path length ($x$-axis). Experiments were conducted in 25 $\times$ 25 rectangular blocks (first row), rectangular mazes (second row), and zig-zag mazes (third row), respectively. The performance using different task representations is highlighted from left to right as a) Naive enumeration, b) Code representation, and c) Grid representation.
  • Figure 4: Path planning performance achieved on 15$\times$15 zig-zag mazes environments. In cases where only the optimal rate is shown, the exact match and optimal rate values are identical.