Risk-Sensitive Exponential Actor Critic

TL;DR

This work tackles unsafe behavior in model-free reinforcement learning by optimizing the entropic risk measure $J^\beta(\pi_\theta) = \frac{1}{\beta} \log \mathbb{E}_{p_\pi(\tau)}[e^{\beta \sum_t r_t}]$, which encodes risk preferences through $\beta$. It derives risk-sensitive policy gradient theorems for on- and off-policy settings and introduces rsEAC, a practical TD3-style actor-critic that avoids explicit exponential value representations by leveraging a log-domain critic $Q_\psi$ with a reparameterized target $Z_\psi = e^{Q_\psi}$, plus gradient normalization and clipping to stabilize learning. The method combines off-policy gradients with stabilization techniques (batch normalization on the critic's last layer and exponent clipping) to produce numerically robust updates, enabling reliable risk-sensitive policy learning in challenging continuous tasks. Empirically, rsEAC yields high-return, risk-aware policies across GridWorld, an inverted-pendulum variant, and risk-modified MuJoCo benchmarks, consistently outperforming or matching strong baselines like R-AC and MVPI while avoiding the numerical pitfalls typical of exponential TD learning.

Abstract

Model-free deep reinforcement learning (RL) algorithms have achieved tremendous success on a range of challenging tasks. However, safety concerns remain when these methods are deployed on real-world applications, necessitating risk-aware agents. A common utility for learning such risk-aware agents is the entropic risk measure, but current policy gradient methods optimizing this measure must perform high-variance and numerically unstable updates. As a result, existing risk-sensitive model-free approaches are limited to simple tasks and tabular settings. In this paper, we provide a comprehensive theoretical justification for policy gradient methods on the entropic risk measure, including on- and off-policy gradient theorems for the stochastic and deterministic policy settings. Motivated by theory, we propose risk-sensitive exponential actor-critic (rsEAC), an off-policy model-free approach that incorporates novel procedures to avoid the explicit representation of exponential value functions and their gradients, and optimizes its policy w.r.t the entropic risk measure. We show that rsEAC produces more numerically stable updates compared to existing approaches and reliably learns risk-sensitive policies in challenging risky variants of continuous tasks in MuJoCo.

Figures (7)

• Figure 1: Instabilities in value functions and gradients. We train value functions for DQN and $Z_\psi(s_t,a_t)$ for a range of $\beta$ settings ($-1, -0.1, 0.1, 1$). We plot the average log-value estimates (left), the L2 gradient norm in log-domain (center), and the average return (right) for 20 episodes over 200k environment steps. For $\beta < 0$, we observe that the value estimates and the gradient norm are almost zero, leading to unstable learning. For $\beta > 0$, the value estimates and gradients are orders of magnitude larger than DQN's estimates. These estimates are propagated by the gradients, resulting in exploding gradients.
• Figure 2: Stable value functions and gradients. We train value functions for DQN and $Q_\psi(s_t,a_t)$ using the normalized clipped gradient for a range of $\beta$ settings ($-1, -0.1, 0.1, 1$). We plot the average value estimates (left), the L2 gradient norm in log-domain (center), and the average return (right) for 20 episodes over 200k environment steps.
• Figure 3: Tabular Risk-Sensitive Policies. Optimal value functions $Z^*(s_t)$ (in log-domain) and corresponding trajectory for different $\beta$ values. Agents trained with negative $\beta$ values learn risk-averse policies that avoid the cliff-region, while agents trained with positive $\beta$ values tend to be risk-seeking and hug the cliff closely. We also recover risk-neutral policies when the magnitude of $|\beta|$ is small.
• Figure 4: Stochastic Cliff GridWorld.Left: 2D grid environment with initial-state and goal-state given by blue circle and green star, respectively. The cliff region is given by gray states. Right: We plot the learned optimal value functions (in log-domain) for $\beta = -10$ and a sampled trajectory. The estimated values explode in magnitude, resulting in a policy incapable of reaching the destination.
• Figure 5: Risk-sensitive policies in Inverted Pendulum. We train rsEAC and R-AC for $\beta =1$ and $\beta = -1$. Left: We plot the average return. Right: Percentage of steps on an episode in risky regions. The solid curves correspond to the mean and shaded regions to $\pm$ one standard deviation over 5 random seeds. rsEAC learns, both, risk-seeking and risk-averse policies that achieve high return, while R-AC encounters numerical instabilities and learns poor policies.

Theorems & Definitions (11)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 3.1
• proof
• Theorem 3.2
• proof
• Theorem 3.3
• proof
• Theorem 3.4