Policy Optimization over General State and Action Spaces
Caleb Ju, Guanghui Lan
TL;DR
This work addresses reinforcement learning with general state and action spaces by extending policy mirror descent (PMD) and introducing policy dual averaging (PDA) as scalable, provably convergent algorithms that accommodate function approximation without mandatory policy parameterization. It establishes linear convergence to global optima or sublinear convergence to stationary points under exact policy evaluation, and derives bounds on how policy-evaluation and approximation errors affect convergence in both finite and continuous action spaces. The authors show that PDA, in particular, can be more amenable to function approximation and offers robust performance in diverse RL tasks, including grid-world, Lunar Lander, inverted pendulum, and LQR settings. Empirical results indicate that the proposed methods are competitive with, and in some cases superior to, state-of-the-art RL algorithms, while maintaining theoretical guarantees. Overall, the paper broadens the applicability of policy-gradient-style methods to general-state RL problems with rigorous convergence analysis and practical approximation strategies.
Abstract
Reinforcement learning (RL) problems over general state and action spaces are notoriously challenging. In contrast to the tableau setting, one can not enumerate all the states and then iteratively update the policies for each state. This prevents the application of many well-studied RL methods especially those with provable convergence guarantees. In this paper, we first present a substantial generalization of the recently developed policy mirror descent method to deal with general state and action spaces. We introduce new approaches to incorporate function approximation into this method, so that we do not need to use explicit policy parameterization at all. Moreover, we present a novel policy dual averaging method for which possibly simpler function approximation techniques can be applied. We establish linear convergence rate to global optimality or sublinear convergence to stationarity for these methods applied to solve different classes of RL problems under exact policy evaluation. We then define proper notions of the approximation errors for policy evaluation and investigate their impact on the convergence of these methods applied to general-state RL problems with either finite-action or continuous-action spaces. To the best of our knowledge, the development of these algorithmic frameworks as well as their convergence analysis appear to be new in the literature. Preliminary numerical results demonstrate the robustness of the aforementioned methods and show they can be competitive with state-of-the-art RL algorithms.