Safe Langevin Soft Actor Critic

TL;DR

The paper tackles tail-risk and generalization challenges in SafeRL by introducing Safe Langevin Soft Actor Critic (SL-SAC), which blends three mechanisms: an ensemble of reward critics optimized with adaptive Stochastic Gradient Langevin Dynamics (aSGLD) to promote diverse, robust value estimates; a distributional cost critic based on Implicit Quantile Networks (IQN) to model the cost distribution and enable CVaR-based safety constraints; and a reactive Lagrangian that updates constraint penalties using empirical CVaR of recent episodes. The authors derive theoretical guarantees on CVaR estimation error and show that CVaR-based Lagrange updates yield stronger constraint signals than traditional expected-cost updates. Empirically, SL-SAC achieves state-of-the-art safety-performance trade-offs on Safety-Gymnasium and demonstrates improved safety in autonomous driving scenarios (MetaDrive), with ablations confirming the importance of both the aSGLD reward ensemble and distributional CVaR risk modeling. The method offers practical impact for safety-critical continuous-control tasks by robustly balancing reward and tail-risk under uncertainty, albeit with modest computational overhead from the ensemble and Langevin dynamics.

Abstract

Balancing reward and safety in constrained reinforcement learning remains challenging due to poor generalization from sharp value minima and inadequate handling of heavy-tailed risk distribution. We introduce Safe Langevin Soft Actor-Critic (SL-SAC), a principled algorithm that addresses both issues through parameter-space exploration and distributional risk control. Our approach combines three key mechanisms: (1) Adaptive Stochastic Gradient Langevin Dynamics (aSGLD) for reward critics, promoting ensemble diversity and escape from poor optima; (2) distributional cost estimation via Implicit Quantile Networks (IQN) with Conditional Value-at-Risk (CVaR) optimization for tail-risk mitigation; and (3) a reactive Lagrangian relaxation scheme that adapts constraint enforcement based on the empirical CVaR of episodic costs. We provide theoretical guarantees on CVaR estimation error and demonstrate that CVaR-based Lagrange updates yield stronger constraint violation signals than expected-cost updates. On Safety-Gymnasium benchmarks, SL-SAC achieves the lowest cost in 7 out of 10 tasks while maintaining competitive returns, with cost reductions of 19-63% in velocity tasks compared to state-of-the-art baselines.

Figures (12)

• Figure 1: SL-SAC architecture overview. The policy interacts with the environment, storing transitions in replay memory. An ensemble of reward critics is trained via aSGLD, while a distributional cost critic learns quantiles via IQN. The policy optimizes a Lagrangian objective balancing reward, entropy, and CVaR-based safety, with $\lambda$ updated using empirical CVaR of recent episodes.
• Figure 2: Training curves for safety-mujoco velocity tasks. Episode return (left) and cost (right), averaged over 5 seeds (solid line: mean, shaded area: standard deviation). The dashed line indicates the constraint threshold $\beta=25$.
• Figure 3: Training curves for safety-navigation tasks. Episode return (left) and cost (right), averaged over 5 seeds (solid line: mean, shaded area: standard deviation). The dashed line indicates the constraint threshold $\beta=25$.
• Figure 4: Training curves for MetaDrive. Episode return (left) and cost (right), averaged over 5 seeds (solid line: mean, shaded area: standard deviation). The dashed line indicates the constraint threshold $\beta=0.1$.
• Figure 5: Ablation studies: optimizer and CVaR level. Left: Reward critic optimizer (aSGLD vs AdamW). Right: CVaR confidence level $\epsilon \in \{0.2, 0.5, 0.75, 1.0\}$ ($\epsilon=1.0$ is expected cost).

Theorems & Definitions (13)

• Theorem 3.1: CVaR Error Bound via Quantile Estimation Error
• Corollary 3.2: Tail Risk Guarantees: CVaR vs. Expected Cost Constraints
• Theorem 3.3: Constraint Violation Probability under CVaR Constraint
• Lemma 2.1: Contraction in Wasserstein Metric bellemare2017distributional
• Theorem 2.2
• proof
• Corollary 2.3: Safety‑Margin Adjustment for CVaR Constraints
• Corollary 2.4
• proof
• Theorem 2.5: CVaR Provides Stronger Constraint Violation Signals