Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty

Uğurcan Özalp

TL;DR

This work proposes a new algorithm called Stochastic Actor-Critic (STAC) that incorporates temporal (one-step) aleatoric uncertainty-uncertainty arising from stochastic transitions, rewards, and policy-induced variability in Bellman targets-to scale pessimistic bias in temporal-difference updates, rather than relying on epistemic uncertainty.

Abstract

Off-policy actor-critic methods in reinforcement learning train a critic with temporal-difference updates and use it as a learning signal for the policy (actor). This design typically achieves higher sample efficiency than purely on-policy methods. However, critic networks tend to overestimate value estimates systematically. This is often addressed by introducing a pessimistic bias based on uncertainty estimates. Current methods employ ensembling to quantify the critic's epistemic uncertainty-uncertainty due to limited data and model ambiguity-to scale pessimistic updates. In this work, we propose a new algorithm called Stochastic Actor-Critic (STAC) that incorporates temporal (one-step) aleatoric uncertainty-uncertainty arising from stochastic transitions, rewards, and policy-induced variability in Bellman targets-to scale pessimistic bias in temporal-difference updates, rather than relying on epistemic uncertainty. STAC uses a single distributional critic network to model the temporal return uncertainty, and applies dropout to both the critic and actor networks for regularization. Our results show that pessimism based on a distributional critic alone suffices to mitigate overestimation, and naturally leads to risk-averse behavior in stochastic environments. Introducing dropout further improves training stability and performance by means of regularization. With this design, STAC achieves improved computational efficiency using a single distributional critic network.

Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty

TL;DR

This work proposes a new algorithm called Stochastic Actor-Critic (STAC) that incorporates temporal (one-step) aleatoric uncertainty-uncertainty arising from stochastic transitions, rewards, and policy-induced variability in Bellman targets-to scale pessimistic bias in temporal-difference updates, rather than relying on epistemic uncertainty.

Abstract

Off-policy actor-critic methods in reinforcement learning train a critic with temporal-difference updates and use it as a learning signal for the policy (actor). This design typically achieves higher sample efficiency than purely on-policy methods. However, critic networks tend to overestimate value estimates systematically. This is often addressed by introducing a pessimistic bias based on uncertainty estimates. Current methods employ ensembling to quantify the critic's epistemic uncertainty-uncertainty due to limited data and model ambiguity-to scale pessimistic updates. In this work, we propose a new algorithm called Stochastic Actor-Critic (STAC) that incorporates temporal (one-step) aleatoric uncertainty-uncertainty arising from stochastic transitions, rewards, and policy-induced variability in Bellman targets-to scale pessimistic bias in temporal-difference updates, rather than relying on epistemic uncertainty. STAC uses a single distributional critic network to model the temporal return uncertainty, and applies dropout to both the critic and actor networks for regularization. Our results show that pessimism based on a distributional critic alone suffices to mitigate overestimation, and naturally leads to risk-averse behavior in stochastic environments. Introducing dropout further improves training stability and performance by means of regularization. With this design, STAC achieves improved computational efficiency using a single distributional critic network.
Paper Structure (44 sections, 3 theorems, 20 equations, 10 figures, 4 tables, 1 algorithm)

This paper contains 44 sections, 3 theorems, 20 equations, 10 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Aleatoric and Epistemic Uncertainty
  4. Maximum Entropy Actor-Critic Reinforcement Learning
  5. Distributional Reinforcement Learning and Overestimation
  6. Distributional Maximum Entropy Actor-Critic
  7. One-step Distributional Uncertainty
  8. Quantifying Overestimation for Sub-Gaussian Critic Distributions
  9. Stochastic Actor-Critic
  10. Policy Evaluation
  11. Policy Improvement
  12. Other Details and Algorithm Summary
  13. Layer Normalization
  14. Lagged critic for TD target
  15. Automatic temperature tuning
  16. ...and 29 more sections

Key Result

Theorem 3.1

Let given state-action value distribution $\mathcal{Q}\in\mathcal{P}(\mathbb{R}^{\mathcal{S}\times\mathcal{A}})$ be sub-Gaussian with mean $\mu(s,a)$ and variance proxy $\sigma^2(s,a)$ for all state-action pairs, with bounded support. For each sample $Q\sim\mathcal{Q}$,

Figures (10)

  • Figure 1: Evolution of the Pessimistic Distributional Bellman Backup. While $\mathcal{T}^{\pi}_{D}\mathcal{Q}(s, a)$ is overestimated, a corrected backup $\mathcal{T}^{\pi}_{\beta, D}\mathcal{Q}(s, a)$ is closer to $\mathcal{Q}(s, a)$.
  • Figure 2: Episodic score curves of STAC and other algorithms.
  • Figure 3: Average episodic value estimation error of STAC and other algorithms.
  • Figure 4: Position occurrence density heatmap of STAC on RiskyPointMass-v0 for different $\beta$ values. The green circle on the middle represents boundary of danger zone. Episodes are terminated when point mass enters green area on the lower left.
  • Figure 5: Episodic score curves of STAC with varying pessimism ($\beta$) parameter, with actor and critic dropout equal to $0.01$.
  • ...and 5 more figures

Theorems & Definitions (7)

  • Definition 3.1
  • Theorem 3.1: Overestimation quantification for sub-Gaussian critics
  • Corollary 3.1.1: Pessimistic critic target
  • Theorem 3.2: Overestimation bound
  • proof : Proof of Theorem \ref{['thm:overestimation']}
  • proof : Proof of Corollary \ref{['cor:pessimism']}
  • proof : Proof of Theorem \ref{['thm:overestimation_bound']}