Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning

TL;DR

LSAC addresses sample inefficiency and exploration challenges in continuous-control RL by embedding Thompson-sampling-inspired uncertainty estimation into an off-policy actor-critic framework. It introduces Distributional Adaptive Langevin Monte Carlo for posterior sampling of a distributional Q-function, augmented by parallel tempering to capture multimodal posteriors and diffusion-synthesized state-action data refined by Q gradients to augment critic updates. Core innovations include the distributional critic objective $L_\mathcal{Z}(\psi)$ with adaptive aSGLD sampling, multimodal Q posterior exploration via a simplified parallel tempering scheme, and the diffusion Q action gradient mechanism that enhances data diversity while steering synthetic actions toward high-value regions; all are integrated with a Max-Ent policy objective $J_\pi=\sum_i \gamma^i [ r_i + \alpha \mathcal{H}(\pi(\cdot|s_i)) ]$. Empirical results on MuJoCo and the DeepMind Control Suite show LSAC achieving or exceeding the performance of strong baselines, with improved exploration, reduced overestimation bias, and competitive wall-clock efficiency, marking the first successful application of LMC-based Thompson sampling in continuous control with continuous action spaces.

Abstract

Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by the success of Thompson sampling for efficient exploration in RL, we propose a novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distributional Langevin Monte Carlo (LMC) based $Q$ updates, parallel tempering for exploring multiple modes of the posterior of the $Q$ function, and diffusion synthesized state-action samples regularized with $Q$ action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks. Notably, LSAC marks the first successful application of an LMC based Thompson sampling in continuous control tasks with continuous action spaces.

Figures (18)

• Figure 1: Training curves for six MuJoCo continuous control tasks over 1e6 time steps. Results are averaged over a window size of 11 epochs and across 10 seeds. Solid lines represent the median performance, and the shaded regions correspond to 90% confidence interval.
• Figure 2: Sensitivity analysis of different parameters on HalfCheetah-v3 environment. A comparison of LSAC with different bias factors $a$, step sizes $\eta_Q$, and inverse temperature parameters $\beta_Q$.
• Figure 3: Ablation on MuJoCo environments comparing the replacement of the distributional critic component with a standard critic. LSAC with distributional critic is more performant than the variant where standard critic is used.
• Figure 4: Normalized $Q$ bias plots for ablation study of the distributional critic component in LSAC. The $Q$ bias value is estimated using the Monte Carlo return over 1e3 episodes on-policy, starting from states sampled in the replay buffer.
• Figure 5: Ablation study of $Q$ action gradient regularization of synthetic state-action samples on the effect of average return in three MuJoCo environments.

Theorems & Definitions (2)

• Proposition B.1
• proof