Iterative Deployment Improves Planning Skills in LLMs
Augusto B. Corrêa, Yoav Gelberg, Luckeciano C. Melo, Ilia Shumailov, André G. Pereira, Yarin Gal
TL;DR
This work tackles how to boost long-horizon planning in LLMs without external planners or explicit demonstrations by introducing iterative deployment: each generation $M_n$ is deployed, its valid traces (as judged by an external validator $V$) are collected into a growing dataset $\mathcal{T}_{n+1}$, and $M_{n+1}$ is obtained by supervised fine-tuning on this curated data. The authors prove that this SFT-on-valid-traces setup is equivalent to REINFORCE with a binary implicit reward $R(x,y)\in\{0,1\}$, and that mixing on- and off-policy traces yields an importance-weighted REINFORCE update, connecting the mechanism to RL in the outer loop. Empirically, they validate on classical planning benchmarks (Blocksworld, Rovers, Sokoban) using a 4B base model and observe substantial gains: after five deploys, performance more than doubles across domains and longer-horizon plans emerge, indicating out-of-distribution generalization. The paper also discusses AI-safety implications of implicit rewards and biases in external validators, arguing that iterative deployment can offer a viable RL-like training alternative while highlighting the need to study its safety and robustness characteristics.
Abstract
We show that iterative deployment of large language models (LLMs), each fine-tuned on data carefully curated by users from the previous models' deployment, can significantly change the properties of the resultant models. By testing this mechanism on various planning domains, we observe substantial improvements in planning skills, with later models displaying emergent generalization by discovering much longer plans than the initial models. We then provide theoretical analysis showing that iterative deployment effectively implements reinforcement learning (RL) training in the outer-loop (i.e. not as part of intentional model training), with an implicit reward function. The connection to RL has two important implications: first, for the field of AI safety, as the reward function entailed by repeated deployment is not defined explicitly, and could have unexpected implications to the properties of future model deployments. Second, the mechanism highlighted here can be viewed as an alternative training regime to explicit RL, relying on data curation rather than explicit rewards.