Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Can Plan Only If We Tell Them

Bilgehan Sel, Ruoxi Jia, Ming Jin

TL;DR

AoT+ enhances Algorithm-of-Thought prompting with periodic state regeneration, random trace augmentation, and memoization to enable autonomous long-horizon planning by LLMs. The approach yields state-of-the-art results on planning benchmarks like Blocksworld and Logistics, often surpassing human baselines and eliminating dependency on external verifiers. Across multiple LLM families, AoT+ demonstrates robust performance and improved efficiency, suggesting that LLMs possess latent planning capabilities that can be activated through structured prompting. This work has practical implications for building resource-efficient, autonomous planning agents in complex domains.

Abstract

Large language models (LLMs) have demonstrated significant capabilities in natural language processing and reasoning, yet their effectiveness in autonomous planning has been under debate. While existing studies have utilized LLMs with external feedback mechanisms or in controlled environments for planning, these approaches often involve substantial computational and development resources due to the requirement for careful design and iterative backprompting. Moreover, even the most advanced LLMs like GPT-4 struggle to match human performance on standard planning benchmarks, such as the Blocksworld, without additional support. This paper investigates whether LLMs can independently generate long-horizon plans that rival human baselines. Our novel enhancements to Algorithm-of-Thoughts (AoT), which we dub AoT+, help achieve state-of-the-art results in planning benchmarks out-competing prior methods and human baselines all autonomously.

LLMs Can Plan Only If We Tell Them

TL;DR

AoT+ enhances Algorithm-of-Thought prompting with periodic state regeneration, random trace augmentation, and memoization to enable autonomous long-horizon planning by LLMs. The approach yields state-of-the-art results on planning benchmarks like Blocksworld and Logistics, often surpassing human baselines and eliminating dependency on external verifiers. Across multiple LLM families, AoT+ demonstrates robust performance and improved efficiency, suggesting that LLMs possess latent planning capabilities that can be activated through structured prompting. This work has practical implications for building resource-efficient, autonomous planning agents in complex domains.

Abstract

Large language models (LLMs) have demonstrated significant capabilities in natural language processing and reasoning, yet their effectiveness in autonomous planning has been under debate. While existing studies have utilized LLMs with external feedback mechanisms or in controlled environments for planning, these approaches often involve substantial computational and development resources due to the requirement for careful design and iterative backprompting. Moreover, even the most advanced LLMs like GPT-4 struggle to match human performance on standard planning benchmarks, such as the Blocksworld, without additional support. This paper investigates whether LLMs can independently generate long-horizon plans that rival human baselines. Our novel enhancements to Algorithm-of-Thoughts (AoT), which we dub AoT+, help achieve state-of-the-art results in planning benchmarks out-competing prior methods and human baselines all autonomously.
Paper Structure (29 sections, 4 figures, 6 tables)

This paper contains 29 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sequential Decision-Making with LLMs.
  4. Self-Verification.
  5. Classical Search Algorithms.
  6. Prompting Methodologies for Planning Problems
  7. Planning vs. Myopic Problems
  8. The Incompatibility of Chain-of-Thought in Planning Problems
  9. Algorithm-of-Thoughts Prompting for Planning
  10. AoT+ Prompting
  11. Use of Random Solution Traces Does Not Degrade Performance
  12. Memoization Avoids Hallucinations
  13. Experimental Results
  14. Problem Setups
  15. Main Results
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Illustration outlining differences between AoT+ and LLM-Modulo-like frameworks.
  • Figure 2: Observed tendency of state-of-the-art LLMs to make simple arithmetic errors when following Chain of Thought (CoT) prompting.
  • Figure 3: Comparison between AoT and AoT+ for Blocksworld benchmark. Due to AoT's computational overhead of reprocessing the entire context, it hallucinates state and produces an action for another state. AoT+ on the other hand, periodically restates and caches the current problem to hop to any previously visited node.
  • Figure 4: Comparison of error rates in state estimation with respect to solution depth for AoT and AoT+ in the Logistics benchmark using LLaMA 3.1 70B model.