A Two-Timescale Primal-Dual Framework for Reinforcement Learning via Online Dual Variable Guidance
Axel Friedrich Wolter, Tobias Sutter
TL;DR
The paper addresses reinforcement learning under a regularized Markov Decision Process by leveraging a linear programming reformulation and a double-regularized Lagrangian. It introduces PGDA-RL, a two-timescale primal-dual Projected Gradient Descent-Ascent algorithm that uses experience replay and online updates from a single trajectory, guided by the dual occupancy-measure variable. The authors prove almost-sure convergence of the last iterates to the optimal regularized value function $V^*_r$ and the corresponding policy $\\pi^*_r$ under weak assumptions, without requiring a simulator or a fixed behavioral policy, and extend the analysis to asynchronous settings with structured replay buffers. The work combines LP-based RL with mature stochastic approximation theory to yield provable convergence for both synchronous and asynchronous learning and demonstrates empirical viability on a standard tabular task, highlighting the practical impact of dual-variable guidance and two-timescale dynamics for stable RL training.
Abstract
We study reinforcement learning by combining recent advances in regularized linear programming formulations with the classical theory of stochastic approximation. Motivated by the challenge of designing algorithms that leverage off-policy data while maintaining on-policy exploration, we propose PGDA-RL, a novel primal-dual Projected Gradient Descent-Ascent algorithm for solving regularized Markov Decision Processes (MDPs). PGDA-RL integrates experience replay-based gradient estimation with a two-timescale decomposition of the underlying nested optimization problem. The algorithm operates asynchronously, interacts with the environment through a single trajectory of correlated data, and updates its policy online in response to the dual variable associated with the occupation measure of the underlying MDP. We prove that PGDA-RL converges almost surely to the optimal value function and policy of the regularized MDP. Our convergence analysis relies on tools from stochastic approximation theory and holds under weaker assumptions than those required by existing primal-dual RL approaches, notably removing the need for a simulator or a fixed behavioral policy.