Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation

TL;DR

The paper studies entropy-regularized natural policy gradient (NPG) methods with linear function approximation under softmax parameterization in reinforcement learning. It establishes finite-time convergence with averaging, showing that entropy-regularized NPG satisfies the persistence of excitation condition and achieves a fast convergence rate of $\tilde{O}(1/T)$ up to a function-approximation error in regularized MDPs. Notably, the results require no a priori assumptions on the policies. Under mild regularity conditions on the concentrability coefficient and basis vectors, the method exhibits linear convergence up to the function-approximation error.

Abstract

Natural policy gradient (NPG) methods with entropy regularization achieve impressive empirical success in reinforcement learning problems with large state-action spaces. However, their convergence properties and the impact of entropy regularization remain elusive in the function approximation regime. In this paper, we establish finite-time convergence analyses of entropy-regularized NPG with linear function approximation under softmax parameterization. In particular, we prove that entropy-regularized NPG with averaging satisfies the \emph{persistence of excitation} condition, and achieves a fast convergence rate of $\tilde{O}(1/T)$ up to a function approximation error in regularized Markov decision processes. This convergence result does not require any a priori assumptions on the policies. Furthermore, under mild regularity conditions on the concentrability coefficient and basis vectors, we prove that entropy-regularized NPG exhibits \emph{linear convergence} up to a function approximation error.