Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks

Jundong Zhang, Yuhui Situ, Fanji Zhang, Rongji Deng, Tianqi Wei

TL;DR

A reinforcement learning framework that discretizes continuous action spaces to approximate multimodal distributions, employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and introduces a dual-critic architecture for more accurate discrete value distribution estimation is proposed.

Abstract

Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.

D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks

TL;DR

A reinforcement learning framework that discretizes continuous action spaces to approximate multimodal distributions, employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and introduces a dual-critic architecture for more accurate discrete value distribution estimation is proposed.

Abstract

Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.
Paper Structure (30 sections, 1 theorem, 46 equations, 13 figures, 5 tables, 1 algorithm)

This paper contains 30 sections, 1 theorem, 46 equations, 13 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Models & Methods
  4. HRHR scenario
  5. Multidimensional Discrete Actors
  6. Clipped Double Q-learning for Discrete Value Distribution
  7. Critic Learning
  8. Policy Learning
  9. Exploration
  10. Experiments
  11. BipedalWalkerHardcore-v3
  12. FetchPush-v3
  13. Mujoco Environment
  14. Discussion and Conclusion
  15. Appendix
  16. ...and 15 more sections

Key Result

Theorem 1

For Gaussian policy $\pi_\theta(\cdot|s) = \mathcal{N}(\mu_\theta(s), \sigma_\theta^2(s)I)$ in an HRHR scenario at state $s$ with high-reward grains $\mathbb{G}_1$ of diameter $\delta$, if $\sigma_\theta(s) > \delta$, then the gradient update satisfies: where $\Delta\theta_{R_i}$ is the update direction toward region $R_i$, and $J(\theta)$ is the expected return. This implies gradient updates pre

Figures (13)

  • Figure 1: Return and risk are in two dimensions. In an ideal task, higher expected returns are associated with lower-risk actions, as shown by the green line in (a) and panel (b). However, in tasks where favorable outcomes are intertwined with risk, the highest returns have to be extracted among high-risk actions, as shown by the red line in (a) and panel (c). Actions sampled from Gaussian distribution can be hard to capture the high return points during learning as the expectation on the distribution is low, but discrete actions can capture the actions.
  • Figure 2: Clipped Double Discrete Value Distribution
  • Figure 3: Training Curves for BipedalWalkerHardcore-v3 Experiments and FetchPush-v4
  • Figure 4: The Distribution Plots of Reward Returns in BipedalWalkerHardcore-v3. ( Blue representing our algorithm and green representing the C51 algorithm. )
  • Figure 5: Operations to update $Z$. (a) The current value distribution of $Z$. (b) Discount factor $\gamma$ changes the shape in the dimension of atoms. (c) The current reward $R$ shifts the distribution in the dimension of atoms. (d) The resulting distribution $R+\gamma Z$ is mapped back to the original atoms by $\Phi$.
  • ...and 8 more figures

Theorems & Definitions (5)

  • Definition 1: High-Risk-High-Return Scenario
  • Definition 2: High-Reward Grain
  • Theorem 1: Policy Preference in HRHR Scenarios
  • Definition 3
  • proof