List Replicable Reinforcement Learning
Bohan Zhang, Michael Chen, A. Pavan, N. V. Vinodchandran, Lin F. Yang, Ruosong Wang
TL;DR
The paper tackles instability and replicability in reinforcement learning by introducing list replicability within the PAC-RL framework, distinguishing weak and strong forms. It develops two main theoretical pillars: a black-box reduction yielding weakly k-list replicable policies with polynomial list size, and a strongly k-list replicable online algorithm with polynomial sample and trace complexity. Central to the approach is a robust planning technique that uses a random tolerance to bound policy variability and a reachability-based truncation scheme to preserve replicability, enabling controlled, near-optimal outputs. Empirical results across toy and standard RL benchmarks demonstrate the practical stability benefits of incorporating the robust planner into existing RL frameworks, suggesting a viable path to more reliable RL deployments.
Abstract
Replicability is a fundamental challenge in reinforcement learning (RL), as RL algorithms are empirically observed to be unstable and sensitive to variations in training conditions. To formally address this issue, we study \emph{list replicability} in the Probably Approximately Correct (PAC) RL framework, where an algorithm must return a near-optimal policy that lies in a \emph{small list} of policies across different runs, with high probability. The size of this list defines the \emph{list complexity}. We introduce both weak and strong forms of list replicability: the weak form ensures that the final learned policy belongs to a small list, while the strong form further requires that the entire sequence of executed policies remains constrained. These objectives are challenging, as existing RL algorithms exhibit exponential list complexity due to their instability. Our main theoretical contribution is a provably efficient tabular RL algorithm that guarantees list replicability by ensuring the list complexity remains polynomial in the number of states, actions, and the horizon length. We further extend our techniques to achieve strong list replicability, bounding the number of possible policy execution traces polynomially with high probability. Our theoretical result is made possible by key innovations including (i) a novel planning strategy that selects actions based on lexicographic order among near-optimal choices within a randomly chosen tolerance threshold, and (ii) a mechanism for testing state reachability in stochastic environments while preserving replicability. Finally, we demonstrate that our theoretical investigation sheds light on resolving the \emph{instability} issue of RL algorithms used in practice. In particular, we show that empirically, our new planning strategy can be incorporated into practical RL frameworks to enhance their stability.