Pure Planning to Pure Policies and In Between with a Recursive Tree Planner

TL;DR

This work tackles the challenge of unifying planning with learned policies by introducing the Recursive Tree Planner (RTP) within the Plan Learn Plan (PLP) framework. RTP enables a spectrum from pure planning to pure policy execution through hierarchical search, near-greedy exploration, generalized actions, boundary-state exploration, and the ability to return multiple sub-goals. The authors formalize discrete and continuous problem formulations, provide an informal optimality discussion, and demonstrate zero-shot transfer across problem classes using learned policies, plus simultaneous multi-level policy learning across a variety of Box2D and MuJoCo tasks. The results show substantial speedups and feasible solutions for long-horizon problems, with notable zero-shot transfer capabilities and robust improvements when combining planning and learning. Overall, RTP/PLP offers a scalable, domain-agnostic approach to integrating model-based planning with data-driven policies, promising practical impact for robotics and autonomous systems.

Abstract

A recursive tree planner (RTP) is designed to function as a pure planner without policies at one extreme and run a pure greedy policy at the other. In between, the RTP exploits policies to improve planning performance and improve zero-shot transfer from one class of planning problem to another. Policies are learned through imitation of the planner. These are then used by the planner to improve policies in a virtuous cycle. To improve planning performance and zero-shot transfer, the RTP incorporates previously learned tasks as generalized actions (GA) at any level of its hierarchy, and can refine those GA by adding primitive actions at any level too. For search, the RTP uses a generalized Dijkstra algorithm [Dijkstra 1959] which tries the greedy policy first and then searches over near-greedy paths and then farther away as necessary. The RPT can return multiple sub-goals from lower levels as well as boundary states near obstacles, and can exploit policies with background and object-number invariance. Policies at all levels of the hierarchy can be learned simultaneously or in any order or come from outside the framework. The RTP is tested here on a variety of Box2d [Cato 2022] problems, including the classic lunar lander [Farama 2022], and on the MuJoCo [Todorov et al 2012] inverted pendulum.

Figures (15)

• Figure 1: Plan Learn Plan (PLP) framework: planner data from class $C_i$ is used to learn policies $P^{(0)} \to P^{(1)}$. Bootstrapping continues to improve the policies for class $C_i$ when $C_j=C_i$, or policies can be used for zero-shot solutions to $C_j$. Learning can continue to improve the policies for $C_j$ too. $P^{(0)}$ can be non-learned initial search strategies for pure planning.
• Figure 2: Pseudocode for the non-hierarchical planner. Line 4 implements near-greedy search. If a next state $s'$ is already in $T$ and adding it to $s$ improves its $R(s')$, line 15, then it is moved along with its subtree to $s$, lines 16-23.
• Figure 3: Examples of optimal planning solutions using the TP, Section \ref{['TP']}. (a) Optimal path for random negative rewards, shown and random choice of next action and state. (b) Optimal path for uniform positive rewards using the TP with breadth first.
• Figure 4: Pseudocode for the hierarchical planner, Section \ref{['RTP']}. Key differences from the TP are the recursive call to the planner itself, line 9, and the return of multiple states, line27, including subgoals in $\tilde{a}.S_g$ and boundary states $S_B$.
• Figure 5: Box2d robot problems. (a) Simple $grab$, learned from scenes with just one object. (b) Simple $place$ in box learned from scenes with 1-3 objects (c) $place$ with obstacles and many boxes can be solve easily by planner using the 1-3 box policy from (b) by performing a near greedy search.