Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space

Thanh-Tuan Tran, Thanh Nguyen Canh, Nak Young Chong, Xiem HoangVan

TL;DR

This paper conducts a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants of TD3, and introduces a weighted clipped Q-learning target that marginalizes over the discrete action distribution.

Abstract

Reinforcement learning in discrete-continuous hybrid action spaces presents fundamental challenges for robotic manipulation, where high-level task decisions and low-level joint-space execution must be jointly optimized. Existing approaches either discretize continuous components or relax discrete choices into continuous approximations, which suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization. In this paper, we propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that natively handles parameterized hybrid action spaces in a principled manner. We conduct a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants. Building on this analysis, we introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization while improving policy smoothness. Experimental results demonstrate that Hybrid TD3 achieves superior training stability and competitive performance against state-of-the-art hybrid action baselines

Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space

TL;DR

This paper conducts a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants of TD3, and introduces a weighted clipped Q-learning target that marginalizes over the discrete action distribution.

Abstract

Reinforcement learning in discrete-continuous hybrid action spaces presents fundamental challenges for robotic manipulation, where high-level task decisions and low-level joint-space execution must be jointly optimized. Existing approaches either discretize continuous components or relax discrete choices into continuous approximations, which suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization. In this paper, we propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that natively handles parameterized hybrid action spaces in a principled manner. We conduct a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants. Building on this analysis, we introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization while improving policy smoothness. Experimental results demonstrate that Hybrid TD3 achieves superior training stability and competitive performance against state-of-the-art hybrid action baselines
Paper Structure (36 sections, 1 theorem, 57 equations, 7 figures, 3 tables)

This paper contains 36 sections, 1 theorem, 57 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Problem formulation
  4. RL baselines for hybrid action spaces
  5. Hybrid TD3: Weighted Clipped Q-Learning
  6. Theoretical Analysis of Overestimation Bias
  7. Hybrid Action RL Formulation
  8. State Representation
  9. Action Representation
  10. Reward & Penalty Design
  11. Training Protocol
  12. Experimental Results
  13. Experimental Setup
  14. Simulation Environment
  15. Baselines
  16. ...and 21 more sections

Key Result

Theorem 2.1

Let $Q^{\pi}$ denote the true $Q$ function under the current target policy $\pi$, and let $Q_{\theta}$ be its neural network approximation with estimation error $\epsilon^{k}_{\phi,\xi}$ at stage $k$, where $\xi \in \{1,2\}$ indexes the critic networks. Assume the errors are independent of the state

Figures (7)

  • Figure 1: TD3 exhibits superior reward performance than SAC, PPO, and DDPG under aggressive domain randomization.
  • Figure 2: Our proposed DRL system deviates from the traditional Markov Decision Process (MDP) that not only relies on the current trajectory to decide the future but also combines the past trajectories to help the agent learns smoother. This model processes the environment observation $o_t$ that consists of the agent's proprioceptive, exteroceptive, relational, and historical data.
  • Figure 3: Training (left) and test objects (right) used for zero-shot generalization evaluation. Test objects differ substantially from training objects in shape, size, color, and surface texture, constituting a meaningful out-of-distribution evaluation.
  • Figure 4: Estimation bias of the baselines (top row), estimation bias of the proposed methods (middle row), and average return (bottom row) across four manipulation tasks. Solid curves represent mean performance, while shaded areas indicate standard deviations over four independent random seeds.
  • Figure 5: Average return learning curves across four manipulation tasks.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Theorem 2.1