Query-Efficient Planning with Language Models

TL;DR

This work tackles query-efficient planning in environments with expensive world-model interactions by leveraging large language models in two distinct ways. The authors introduce ToI, where an LLM serves as a heuristic within a traditional planner, and Boomerang, a generative LLM planner that proposes whole action sequences and updates its internal model via feedback from the true world model. Empirical results across PlanBench Blocksworld, Logistics, Grippers, and the Robotouille domain show Boomerang achieving the highest efficiency and success under tight query budgets, while ToI variants lag behind in many cases. The paper also links Boomerang to lazy search and posterior sampling theory, and provides ablations demonstrating how prompt engineering and feedback integration improve performance. These findings highlight the practical potential of generative LLM planning with adaptive feedback for real-time, query-constrained planning tasks, with code available online.

Abstract

Planning in complex environments requires an agent to efficiently query a world model to find a feasible sequence of actions from start to goal. Recent work has shown that Large Language Models (LLMs), with their rich prior knowledge and reasoning capabilities, can potentially help with planning by searching over promising states and adapting to feedback from the world. In this paper, we propose and study two fundamentally competing frameworks that leverage LLMs for query-efficient planning. The first uses LLMs as a heuristic within a search-based planner to select promising nodes to expand and propose promising actions. The second uses LLMs as a generative planner to propose an entire sequence of actions from start to goal, query a world model, and adapt based on feedback. We show that while both approaches improve upon comparable baselines, using an LLM as a generative planner results in significantly fewer interactions. Our key finding is that the LLM as a planner can more rapidly adapt its planning strategies based on immediate feedback than LLM as a heuristic. We present evaluations and ablations on Robotouille and PDDL planning benchmarks and discuss connections to existing theory on query-efficient planning algorithms. Code is available at https://github.com/portal-cornell/llms-for-planning

Figures (13)

• Figure 1: Overview of LLM planning methods that find a feasible path with minimal queries to a world model. ReAct selects actions only and must backtrack to undo its actions. ReAct-Select selects both states and actions, allowing it to immediately teleport to better states. Tree of Interaction (ToI) uses a planner to drive the search while using an LLM as a heuristic to select states. Boomerang generates an entire plan, allowing it to immediately switch to a new plan.
• Figure 2: Success of approaches that efficiently reached the goal within 20 world model queries. The classical planner is the best from Appendix \ref{['app:classical_planners_vs_boomerang']}.
• Figure 3: Histogram of interactive approaches' world model queries on Blocksworld problems. Count represents the number of runs that made a specific number of queries (total of 600 runs). Failures are capped at 20 world model queries.
• Figure 4: Optimality rate of interactive approaches that reach the goal within 20 world model queries
• Figure 5: Comparison of ToI-DFS, Boomerang, and Classical planners on Logistics, Grippers, and Robotouille: We chart the success rate given various world model query budgets and observed that Boomerang is most query-efficient at reaching the goal. The applicable classical planners are the best from Appendix \ref{['app:classical_planners_vs_boomerang']}.

Theorems & Definitions (1)

• Theorem 1