Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of actor-critic for entropy regularised MDPs in general action spaces

Denis Zorba, David Šiška, Lukasz Szpruch

TL;DR

The paper addresses convergence and stability of entropy-regularised MDPs in general action spaces using a two-timescale actor–critic scheme, where TD updates govern the critic and mirror-descent updates govern the policy, within a Fisher–Rao gradient-flow framework. Under linear $Q$-realisability, it derives a Lyapunov-based stability analysis and proves exponential convergence to the entropy-regularised optimum up to the critic error, with convergence rates governed by the entropy parameter $ au$ and the timescale separation $ ext{η}_t$. The results illuminate how entropy regularisation interacts with two-time-scale dynamics to ensure stable learning in continuous spaces, and they provide rigorous guarantees for the approximate Fisher–Rao policy flow coupled with TD-based critic updates. Limitations include the continuous-time setting and linear critic approximation; extending to discrete-time updates and non-linear function approximators remains an important direction for future work. Overall, the work advances theoretical understanding of globally convergent actor–critic methods in entropy-regularised MDPs with general action spaces.

Abstract

We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. We demonstrate stability and exponential convergence of the actor critic flow to the optimal policy. Finally, we address the interplay of the timescale separation and entropy regularisation and its effect on stability and convergence.

Convergence of actor-critic for entropy regularised MDPs in general action spaces

TL;DR

The paper addresses convergence and stability of entropy-regularised MDPs in general action spaces using a two-timescale actor–critic scheme, where TD updates govern the critic and mirror-descent updates govern the policy, within a Fisher–Rao gradient-flow framework. Under linear -realisability, it derives a Lyapunov-based stability analysis and proves exponential convergence to the entropy-regularised optimum up to the critic error, with convergence rates governed by the entropy parameter and the timescale separation . The results illuminate how entropy regularisation interacts with two-time-scale dynamics to ensure stable learning in continuous spaces, and they provide rigorous guarantees for the approximate Fisher–Rao policy flow coupled with TD-based critic updates. Limitations include the continuous-time setting and linear critic approximation; extending to discrete-time updates and non-linear function approximators remains an important direction for future work. Overall, the work advances theoretical understanding of globally convergent actor–critic methods in entropy-regularised MDPs with general action spaces.

Abstract

We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. We demonstrate stability and exponential convergence of the actor critic flow to the optimal policy. Finally, we address the interplay of the timescale separation and entropy regularisation and its effect on stability and convergence.
Paper Structure (28 sections, 19 theorems, 163 equations)

This paper contains 28 sections, 19 theorems, 163 equations.

Table of Contents

  1. Introduction
  2. Related works
  3. Our Contribution
  4. Notation
  5. Entropy Regularised Markov Decision Processes
  6. Mirror-Descent and the Fisher-Rao Gradient flow
  7. Actor Critic Methods
  8. Dynamics
  9. Stability
  10. Convergence
  11. Limitations
  12. Acknowledgements
  13. Technical details
  14. Proof of Lemma \ref{['lemma:gradient']}
  15. Proof of Lemma \ref{['lemma:occupancy_prop']}
  16. ...and 13 more sections

Key Result

Theorem 1.1

Let $\tau > 0$. The optimal value function $V^{*}_{\tau}$ is the unique bounded solution of the following Bellman equation: where $Q^*_{\tau}\in B_b(S\times A)$ is defined by Moreover, there is an optimal policy $\pi^*_{\tau} \in \mathcal{P}(A|S)$ given by Finally, the value function $V^{\pi}_{\tau}$ is the unique bounded solution of the following Bellman equation for all $s \in S$

Theorems & Definitions (36)

  • Theorem 1.1: Dynamical Programming Principle
  • Definition 1.1: Admissible Policies
  • Lemma 1.1: Performance difference
  • Definition 4.1
  • Lemma 4.1
  • Lemma 5.1
  • Lemma 5.2
  • Theorem 5.1
  • Corollary 5.1: Stability
  • Corollary 5.2
  • ...and 26 more