Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mirror Descent Actor Critic via Bounded Advantage Learning

Ryo Iwaki

TL;DR

This work addresses continuous-action reinforcement learning with regularized mirror-descent value iteration by introducing Mirror Descent Actor Critic (MDAC), an off-policy actor-critic instantiation that bounds the actor's log-density terms to stabilize learning. The core idea reframes the TD target using bounded soft advantages, yielding a Balancing Advantage Learning (BAL) framework that preserves convergence properties while increasing action gaps to mitigate approximation errors. Theoretical results show BAL can converge under suitable conditions and reduce inherent errors compared to unbounded schemes, while empirical results across MuJoCo, Adroit, and DMC dog demonstrate that MDAC with carefully chosen bounding functions outperforms strong baselines like SAC and TD3. The findings highlight the practical importance of bounding log-density terms in regularized MDVI and suggest robust, scalable improvements for continuous control tasks, with open questions about optimal bounding function design and extension beyond tabular analyses.

Abstract

Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates. Despite its empirical success in discrete action domains and strong theoretical guarantees, the performance of KL-entroy-regularized methods do not surpass a strong entropy-only-regularized method in continuous action domains. In this study, we propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains, and show that its empirical performance is significantly boosted by bounding the actor's log-density terms in the critic's loss function, compared to a non-bounded naive instantiation. Further, we relate MDAC to Advantage Learning by recalling that the actor's log-probability is equal to the regularized advantage function in tabular cases, and theoretically discuss when and why bounding the advantage terms is validated and beneficial. We also empirically explore effective choices for the bounding functions, and show that MDAC performs better than strong non-regularized and entropy-only-regularized methods with an appropriate choice of the bounding functions.

Mirror Descent Actor Critic via Bounded Advantage Learning

TL;DR

This work addresses continuous-action reinforcement learning with regularized mirror-descent value iteration by introducing Mirror Descent Actor Critic (MDAC), an off-policy actor-critic instantiation that bounds the actor's log-density terms to stabilize learning. The core idea reframes the TD target using bounded soft advantages, yielding a Balancing Advantage Learning (BAL) framework that preserves convergence properties while increasing action gaps to mitigate approximation errors. Theoretical results show BAL can converge under suitable conditions and reduce inherent errors compared to unbounded schemes, while empirical results across MuJoCo, Adroit, and DMC dog demonstrate that MDAC with carefully chosen bounding functions outperforms strong baselines like SAC and TD3. The findings highlight the practical importance of bounding log-density terms in regularized MDVI and suggest robust, scalable improvements for continuous control tasks, with open questions about optimal bounding function design and extension beyond tabular analyses.

Abstract

Regularization is a core component of recent Reinforcement Learning (RL) algorithms. Mirror Descent Value Iteration (MDVI) uses both Kullback-Leibler divergence and entropy as regularizers in its value and policy updates. Despite its empirical success in discrete action domains and strong theoretical guarantees, the performance of KL-entroy-regularized methods do not surpass a strong entropy-only-regularized method in continuous action domains. In this study, we propose Mirror Descent Actor Critic (MDAC) as an actor-critic style instantiation of MDVI for continuous action domains, and show that its empirical performance is significantly boosted by bounding the actor's log-density terms in the critic's loss function, compared to a non-bounded naive instantiation. Further, we relate MDAC to Advantage Learning by recalling that the actor's log-probability is equal to the regularized advantage function in tabular cases, and theoretically discuss when and why bounding the advantage terms is validated and beneficial. We also empirically explore effective choices for the bounding functions, and show that MDAC performs better than strong non-regularized and entropy-only-regularized methods with an appropriate choice of the bounding functions.

Paper Structure

This paper contains 38 sections, 21 theorems, 84 equations, 17 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Works.
  3. Contibutions.
  4. PRELIMINARY
  5. MDP and Approximate Value Iteration.
  6. Regularized MDP and MDVI.
  7. MIRROR DESCENT ACTOR CRITIC WITH BOUNDED BONUS TERMS
  8. ANALYSIS
  9. Bounded Advantage Learning
  10. Asymptotic Convergence
  11. Inherent Error Reduction
  12. EXPERIMENT
  13. Mujoco Locomoation Environments
  14. Setup and Metrics.
  15. Effect of bounding functions $f$ and $g$.
  16. ...and 23 more sections

Key Result

Proposition 4.1

Consider the sequence $\Psi_{k+1} \vcentcolon= \mathcal{T}^{fg}_{\pi_{k+1}} \Psi_{k}$ with $\Psi_0 \in \mathbb{R}^{\mathcal{S}\times\mathcal{A}}$, and let $V_{k} = \mathbb{L}^{\alpha} \Psi_{k}$. Assume that for all $k \in \mathbb{N}$ it holds that where $D_{k+1} \!=\! D_{\mathrm{KL}}(\pi_{k+1}\|\pi_{k})$. Then, the sequence $(V_{k})_{k \in \mathbb{N}}$ converges, and the limit $\tilde{V} = \lim_{

Figures (17)

  • Figure 1:
  • Figure 3: Scale comparison of the variables in loss functions. The means of the variables over the multiple sampled minibatchs are plotted. Left: $\log\sigma_{\rm min}\!=\!-5$, Middle: $\log\sigma_{\rm min}\!=\!-2$, Right: $\log\sigma_{\rm min}\!=\!-5$ with bounding by $\tanh$. Top: comparison in critic loss \ref{['eq:mdac:critic']}, Bottom: comparison in actor and entropy losses \ref{['eq:mdac:actor']} and \ref{['eq:mdac:entropy']}. $\alpha$ is indicated by the right y-axis. Blue shaded areas indicate standard deviations. Light blue shaded areas indicate minimum and maximum values.
  • Figure 4: Empirical study to examine how the choices of the bounding functions $f$, $g$ affect the performance of MDAC.
  • Figure 5: Benchmarking results.
  • Figure 6: Per-environment results for Adroit hand manipulation tasks and DeepMind Control Suite dog domain. Mean test rewards over $10$ independent runs are plotted. The shaded areas indicate 25$\%$ and 75$\%$ percentiles.
  • ...and 12 more figures

Theorems & Definitions (39)

  • Proposition 4.1
  • Proposition 4.2
  • Corollary 4.1
  • Lemma B.1
  • proof
  • Lemma B.2
  • proof
  • Lemma B.3
  • proof
  • Lemma B.4
  • ...and 29 more