Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents

Zergham Ahmed, Kazuki Irie, Joshua B. Tenenbaum, Christopher J. Bates, Samuel J. Gershman

TL;DR

The paper addresses efficient hierarchical planning by learning reusable abstractions instead of relying on hand-crafted domain knowledge. It introduces TheoryCoder-2, an autonomous TBRL agent that synthesizes high-level PDDL abstractions and ground-level Python world models via in-context learning and a curriculum of tasks. Empirical results across VGDL-based domains (Sokoban, Labyrinth, Maze), BabyAI, and Minihack show substantial improvements in sample efficiency and generalization, with abstractions that transfer across tasks and environments. This work demonstrates a step toward scalable, human-like abstraction learning in planning agents, reducing human input while enabling broad applicability.

Abstract

Humans learn abstractions and use them to plan efficiently to quickly generalize across tasks -- an ability that remains challenging for state-of-the-art large language model (LLM) agents and deep reinforcement learning (RL) systems. Inspired by the cognitive science of how people form abstractions and intuitive theories of their world knowledge, Theory-Based RL (TBRL) systems, such as TheoryCoder, exhibit strong generalization through effective use of abstractions. However, they heavily rely on human-provided abstractions and sidestep the abstraction-learning problem. We introduce TheoryCoder-2, a new TBRL agent that leverages LLMs' in-context learning ability to actively learn reusable abstractions rather than relying on hand-specified ones, by synthesizing abstractions from experience and integrating them into a hierarchical planning process. We conduct experiments on diverse environments, including BabyAI, Minihack and VGDL games like Sokoban. We find that TheoryCoder-2 is significantly more sample-efficient than baseline LLM agents augmented with classical planning domain construction, reasoning-based planning, and prior program-synthesis agents such as WorldCoder. TheoryCoder-2 is able to solve complex tasks that the baselines fail, while only requiring minimal human prompts, unlike prior TBRL systems.

Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents

TL;DR

The paper addresses efficient hierarchical planning by learning reusable abstractions instead of relying on hand-crafted domain knowledge. It introduces TheoryCoder-2, an autonomous TBRL agent that synthesizes high-level PDDL abstractions and ground-level Python world models via in-context learning and a curriculum of tasks. Empirical results across VGDL-based domains (Sokoban, Labyrinth, Maze), BabyAI, and Minihack show substantial improvements in sample efficiency and generalization, with abstractions that transfer across tasks and environments. This work demonstrates a step toward scalable, human-like abstraction learning in planning agents, reducing human input while enabling broad applicability.

Abstract

Humans learn abstractions and use them to plan efficiently to quickly generalize across tasks -- an ability that remains challenging for state-of-the-art large language model (LLM) agents and deep reinforcement learning (RL) systems. Inspired by the cognitive science of how people form abstractions and intuitive theories of their world knowledge, Theory-Based RL (TBRL) systems, such as TheoryCoder, exhibit strong generalization through effective use of abstractions. However, they heavily rely on human-provided abstractions and sidestep the abstraction-learning problem. We introduce TheoryCoder-2, a new TBRL agent that leverages LLMs' in-context learning ability to actively learn reusable abstractions rather than relying on hand-specified ones, by synthesizing abstractions from experience and integrating them into a hierarchical planning process. We conduct experiments on diverse environments, including BabyAI, Minihack and VGDL games like Sokoban. We find that TheoryCoder-2 is significantly more sample-efficient than baseline LLM agents augmented with classical planning domain construction, reasoning-based planning, and prior program-synthesis agents such as WorldCoder. TheoryCoder-2 is able to solve complex tasks that the baselines fail, while only requiring minimal human prompts, unlike prior TBRL systems.
Paper Structure (15 sections, 3 figures, 1 table, 1 algorithm)

This paper contains 15 sections, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Theory-based Reinforcement Learning
  4. TheoryCoder
  5. Method: TheoryCoder-2
  6. Learning Abstractions
  7. Reusing and Growing the Library of Abstractions
  8. Experiments
  9. Evaluating Abstraction Learning and Reuse in Simple Environments
  10. Transferring abstractions to harder problems
  11. Discussion
  12. Related Work
  13. Conclusion
  14. Language Model Prompts
  15. Results and Outputs

Figures (3)

  • Figure 1: Comparison of agent–environment interaction between methods. WorldCoder and LLM + P both fall under the LLM + Planner category.
  • Figure 2: An illustration of the curriculum used in our experiments. A curriculum is a sequence of episodes in which each episode contains one or more environments/games. The sequence of the first episode (Labyrinth) and the second one (Maze, and Sokoban) is studied in Experiment 1 (Sec. \ref{['sec:exp1']}), while the entire sequence is used in Experiment 2 (Sec. \ref{['sec:exp3']}). In experiment 3 (Minihack), we use a separate curriculum. The blue arrows indicate the abstractions that TheoryCoder-2 learned.
  • Figure 3: Success rate as a function of compute time, averaged across all games. The TC Family represents TheoryCoder-2 and its variants. TheoryCoder and its ablations are able to solve more tasks with less compute time than the reasoning models that use high reasoning effort. LLM + $\pi$ is shown with three different reasoning efforts.