Finite-time Convergence Analysis of Actor-Critic with Evolving Reward
Rui Hu, Yu Chen, Longbo Huang
TL;DR
The authors address finite-time convergence of a single-timescale actor-critic algorithm when rewards evolve over time under Markovian sampling. They develop non-asymptotic bounds for actor and critic errors and show that an $O\left(\frac{1}{\sqrt{T}}\right)$ rate is preserved if reward changes are slow, including when the reward evolves via a gradient-based update on the same timescale. A key technical contribution is a Lipschitz-based analysis of how evolving rewards affect the policy objective and a novel bound on distribution mismatch due to Markovian sampling, which tightens convergence guarantees beyond prior static-reward results. The results provide theoretical support for common evolving-reward RL techniques such as reward shaping, entropy regularization, and curriculum learning, and yield practical guidance (e.g., gradient clipping) for maintaining stability. This work lays the groundwork for extending finite-time analyses to nonlinear function approximators and more complex dynamic objectives in reinforcement learning.
Abstract
Many popular practical reinforcement learning (RL) algorithms employ evolving reward functions-through techniques such as reward shaping, entropy regularization, or curriculum learning-yet their theoretical foundations remain underdeveloped. This paper provides the first finite-time convergence analysis of a single-timescale actor-critic algorithm in the presence of an evolving reward function under Markovian sampling. We consider a setting where the reward parameters may change at each time step, affecting both policy optimization and value estimation. Under standard assumptions, we derive non-asymptotic bounds for both actor and critic errors. Our result shows that an $O(1/\sqrt{T})$ convergence rate is achievable, matching the best-known rate for static rewards, provided the reward parameters evolve slowly enough. This rate is preserved when the reward is updated via a gradient-based rule with bounded gradient and on the same timescale as the actor and critic, offering a theoretical foundation for many popular RL techniques. As a secondary contribution, we introduce a novel analysis of distribution mismatch under Markovian sampling, improving the best-known rate by a factor of $\log^2T$ in the static-reward case.