Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success
Daniel Russo
TL;DR
This work treats success conditioning as an exact, conservative policy-improvement operator and proves it solves a trust-region optimization with a chi-squared divergence budget, where the radius is automatically set by action-influence. It establishes a precise identity linking first-order improvement, the magnitude of policy change, and action-influence at every state, explaining when success conditioning yields meaningful gains and why its failure manifests as minimal policy movement. The analysis contrasts the chi-squared geometry with KL-based TRPO/MPO-type methods, showing both the safety guarantees and the automatic, data-driven radius; it also examines the implications for offline learning, including deployment-proximity bounds. The paper further discusses return-thresholding as a proxy reward, detailing when it amplifies improvement and when misalignment can degrade true performance, and situates success conditioning within broader literatures on return-conditioned RL and control-as-inference. Overall, the results clarify why success conditioning is a conservative yet broadly applicable mechanism for policy improvement, with practical guidance on when and how to apply it and where its limitations lie.
Abstract
A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names -- rejection sampling with SFT, goal-conditioned RL, Decision Transformers -- yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a $χ^2$ divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence -- measuring how random variation in action choices affects success rates -- are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing this can amplify improvement, but at the cost of potential misalignment with the true objective.