On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization
Ling Liang, Haizhao Yang
TL;DR
This work studies a general, regularized reward optimization problem in reinforcement learning, encompassing decision-dependent distributions. It analyzes a stochastic proximal gradient method and demonstrates an $O(\varepsilon^{-4})$ sample complexity to reach an $\varepsilon$-stationary point; it then introduces a variance-reduced PAGE-based estimator to reduce this to $O(\varepsilon^{-3})$ under additional assumptions, aligning with state-of-the-art results for discounted MDPs. The approach leverages Lipschitz smoothness of the objective $\mathcal{J}(\theta)$ and a proximal structure $\mathcal{G}(\theta)$, with an emphasis on non-oblivious RL settings and importance-weighted gradient estimation. The findings contribute a novel, theoretically-grounded variance-reduction pathway for general reward optimization in RL and offer insights into efficient sample use for proximal-gradient-based RL algorithms.
Abstract
We consider a regularized expected reward optimization problem in the non-oblivious setting that covers many existing problems in reinforcement learning (RL). In order to solve such an optimization problem, we apply and analyze the classical stochastic proximal gradient method. In particular, the method has shown to admit an $O(ε^{-4})$ sample complexity to an $ε$-stationary point, under standard conditions. Since the variance of the classical stochastic gradient estimator is typically large, which slows down the convergence, we also apply an efficient stochastic variance-reduce proximal gradient method with an importance sampling based ProbAbilistic Gradient Estimator (PAGE). Our analysis shows that the sample complexity can be improved from $O(ε^{-4})$ to $O(ε^{-3})$ under additional conditions. Our results on the stochastic (variance-reduced) proximal gradient method match the sample complexity of their most competitive counterparts for discounted Markov decision processes under similar settings. To the best of our knowledge, the proposed methods represent a novel approach in addressing the general regularized reward optimization problem.