Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Code-Driven Planning in Grid Worlds with Large Language Models

Ashwath Vaithinathan Aravindan, Zhisheng Tang, Mayank Kejriwal

TL;DR

The paper introduces Iterative Programmatic Planning (IPP), a framework that uses large language models to synthesize executable Python policies for grid-world planning by treating planning as code generation. IPP integrates direct generation, pseudocode-conditioned refinement, and curriculum prompting within an Iterative Refinement loop that uses task-performance feedback to improve the policy without task-specific training. Across GRASP and MiniGrid benchmarks and six state-of-the-art LLMs, IPP with Iterative Refinement consistently surpasses direct code generation and non-code prompting baselines, achieving state-of-the-art results on GRASP and substantial gains on MiniGrid while keeping costs amortizable due to reusable code. The framework emphasizes transparency, generalization, and efficiency, offering a practical alternative to reinforcement learning for planning in structured environments. Limitations include variability across models in response to IR and the non-gradient-based nature of the refinement process, suggesting avenues for further integration with execution feedback and safety considerations.

Abstract

We propose an iterative programmatic planning (IPP) framework for solving grid-based tasks by synthesizing interpretable agent policies expressed in code using large language models (LLMs). Instead of relying on traditional search or reinforcement learning, our approach uses code generation as policy synthesis, where the LLM outputs executable programs that map environment states to action sequences. Our proposed architecture incorporates several prompting strategies, including direct code generation, pseudocode-conditioned refinement, and curriculum-based prompting, but also includes an iterative refinement mechanism that updates code based on task performance feedback. We evaluate our approach using six leading LLMs and two challenging grid-based benchmarks (GRASP and MiniGrid). Our IPP framework demonstrates improvements over direct code generation ranging from 10\% to as much as 10x across five of the six models and establishes a new state-of-the-art result for GRASP. IPP is found to significantly outperform direct elicitation of a solution from GPT-o3-mini (by 63\% on MiniGrid to 116\% on GRASP), demonstrating the viability of the overall approach. Computational costs of all code generation approaches are similar. While code generation has a higher initial prompting cost compared to direct solution elicitation (\$0.08 per task vs. \$0.002 per instance for GPT-o3-mini), the code can be reused for any number of instances, making the amortized cost significantly lower (by 400x on GPT-o3-mini across the complete GRASP benchmark).

Code-Driven Planning in Grid Worlds with Large Language Models

TL;DR

The paper introduces Iterative Programmatic Planning (IPP), a framework that uses large language models to synthesize executable Python policies for grid-world planning by treating planning as code generation. IPP integrates direct generation, pseudocode-conditioned refinement, and curriculum prompting within an Iterative Refinement loop that uses task-performance feedback to improve the policy without task-specific training. Across GRASP and MiniGrid benchmarks and six state-of-the-art LLMs, IPP with Iterative Refinement consistently surpasses direct code generation and non-code prompting baselines, achieving state-of-the-art results on GRASP and substantial gains on MiniGrid while keeping costs amortizable due to reusable code. The framework emphasizes transparency, generalization, and efficiency, offering a practical alternative to reinforcement learning for planning in structured environments. Limitations include variability across models in response to IR and the non-gradient-based nature of the refinement process, suggesting avenues for further integration with execution feedback and safety considerations.

Abstract

We propose an iterative programmatic planning (IPP) framework for solving grid-based tasks by synthesizing interpretable agent policies expressed in code using large language models (LLMs). Instead of relying on traditional search or reinforcement learning, our approach uses code generation as policy synthesis, where the LLM outputs executable programs that map environment states to action sequences. Our proposed architecture incorporates several prompting strategies, including direct code generation, pseudocode-conditioned refinement, and curriculum-based prompting, but also includes an iterative refinement mechanism that updates code based on task performance feedback. We evaluate our approach using six leading LLMs and two challenging grid-based benchmarks (GRASP and MiniGrid). Our IPP framework demonstrates improvements over direct code generation ranging from 10\% to as much as 10x across five of the six models and establishes a new state-of-the-art result for GRASP. IPP is found to significantly outperform direct elicitation of a solution from GPT-o3-mini (by 63\% on MiniGrid to 116\% on GRASP), demonstrating the viability of the overall approach. Computational costs of all code generation approaches are similar. While code generation has a higher initial prompting cost compared to direct solution elicitation (\0.002 per instance for GPT-o3-mini), the code can be reused for any number of instances, making the amortized cost significantly lower (by 400x on GPT-o3-mini across the complete GRASP benchmark).
Paper Structure (29 sections, 3 figures, 7 tables)

This paper contains 29 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Code-driven Planning with Iterative Refinement
  4. Direct Generation
  5. Pseudocode Extension
  6. Step-by-Step
  7. Iterative Refinement
  8. Experimental setup
  9. Evaluation Benchmarks
  10. Prompting Strategies and Baselines
  11. LLMs Evaluated
  12. Results
  13. Discussion and Limitations
  14. Related work
  15. Appendix
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Example of the Unlock Pickup task from the MiniGrid benchmark. The agent (red triangle) must retrieve a box (purple item) located in a separate room behind a locked door (yellow). Directions in the action sequence indicate the direction the agent turns to face before moving. (a) shows the original environment setup (b) shows actions generated by Claude-3.7 using direct generation, which fails and yields a reward of 0. (c) shows iterative refinement, which successfully completes the task with a reward of 0.96.
  • Figure 2: Performance of GPT-o1, Claude-3.7, and GPT-o3-mini across four tasks—GRASP, Unlock, Door-Key, and Unlock-Pickup—measured over successive refinement iterations. For GRASP, the metric is energy collected; for MiniGrid tasks, the metric is reward (maximum = 1.0). Iteration 0 corresponds to the initial output produced by the direct generation method.
  • Figure 3: Left: Performance comparison of Greedy and Pseudocode Extension strategies on the GRASP task, evaluated across GPT-o1, Claude-3.7, and GPT-o3-mini. The results highlight the performance gains enabled by pseudocode-guided reasoning. Right: Step-by-step performance on GRASP, where environment complexity is introduced incrementally over four steps.