Enhancing LLM Planning Capabilities through Intrinsic Self-Critique

TL;DR

The paper tackles improving LLM-based planning by introducing intrinsic self-critique, eliminating the need for external verifiers. It presents a zero-shot or few-shot prompting framework that iteratively generates plans and critiques them using the LLM itself, augmented by self-consistency. Across Blocksworld, Mystery Blocksworld, Logistics, and MiniGrid benchmarks, the method yields substantial accuracy gains and achieves state-of-the-art results for October 2024 model checkpoints, highlighting its model-agnostic applicability. The work demonstrates that intrinsic self-improvement can bridge the gap between LLM planning and traditional planners, with implications for applying self-critique to more complex search strategies and larger models. Overall, the approach provides a practical, scalable pathway to enhance planning reliability in real-world tasks without external verification.

Abstract

We demonstrate an approach for LLMs to critique their \emph{own} answers with the goal of enhancing their performance that leads to significant improvements over established planning benchmarks. Despite the findings of earlier research that has cast doubt on the effectiveness of LLMs leveraging self critique methods, we show significant performance gains on planning datasets in the Blocksworld domain through intrinsic self-critique, without external source such as a verifier. We also demonstrate similar improvements on Logistics and Mini-grid datasets, exceeding strong baseline accuracies. We employ a few-shot learning technique and progressively extend it to a many-shot approach as our base method and demonstrate that it is possible to gain substantial improvement on top of this already competitive approach by employing an iterative process for correction and refinement. We illustrate how self-critique can significantly boost planning performance. Our empirical results present new state-of-the-art on the class of models considered, namely LLM model checkpoints from October 2024. Our primary focus lies on the method itself, demonstrating intrinsic self-improvement capabilities that are applicable regardless of the specific model version, and we believe that applying our method to more complex search techniques and more capable models will lead to even better performance.

Figures (7)

• Figure 1: Illustration of the iterative self-improvement process for Large Language Models (LLMs) using in-context learning to incorporate self-critique feedback. The LLM, also represented by the explorer character, functions as the agent's brain, accepting prompts as inputs and generating outputs, represented by green and red semi-circles, respectively. Each iteration of the self-improvement mechanism comprises two key steps: i) plan generation and ii) self-critiquing, aimed at iteratively refining LLM outputs. In step i), the LLM generates a plan (symbolized by a map) based on a prompt incorporating domain-specific knowledge and instructions (symbolized by the treasure chest). Step ii) involves a self-critique mechanism where the LLM evaluates its own performance, providing correctness assessments and justifications, again leveraging domain knowledge. The process continues until a plan deemed correct is identified. Previous plans and their associated self-critique feedback are aggregated into a collection (symbolized by a bag), serving as contextual material for subsequent plan generation cycles.
• Figure 2: The figure illustrates how number of correct instances increases as we increase the number of Self-Critique iterations on Blocksworld (left), Mini-Grid (center) and Logistics (right). The initial step (step $0$), depicted in the figure, represents the baseline without Self-Critique. Subsequent iterations, from 1 to 10, each involve a Self-Critique and a refinement request to an LLM, demonstrating progressive improvements of correct instances. We use a $16$-shot planning prompt and $0$-shots for the critique prompt on all benchmarks. Model: Gemini 1.5 Pro
• Figure 3: Performance on Blocksworld (left), Mini-Grid (center) and Logistics (right) with increasing number of shots in the planning prompt. While performance on Logistic stagnates using more than 2 shots, the more exemplars in Blocksworld and Minigrid the better. To strike a good balance between LLM context length and accuracy over all benchmarks, we used this experiment to select the number of planning shots, and chose it to be 16 exemplars. Model: Gemini 1.5 Pro
• Figure 4: Accuracy of the Self-Critique process over the course of 10 Self-Critique steps. For these ablations, we use the Blocksworld validation set with 3-7 blocks, a 16-shot plan prediction prompt, and an 8-shot Self-Critique prompt. Overall, we observe relatively high accuracy and recall, but lower precision. Figure \ref{['fig:error_analysis_full']} in Appendix \ref{['ap:ablation:full']} shows this error analysis on more ablations in terms of domain definition, temperature, instructions, number of steps, etc.
• Figure 5: Exploration of planning performance using Gemma 2-27b.