Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method

Andy Wu, Chun-Cheng Lin, Rung-Tzuo Liaw, Yuehua Huang, Chihjung Kuo, Chia Tong Weng

TL;DR

This work introduces Multi-Actor Multi-Critic (MAMC), a deep deterministic RL framework that jointly leverages many actors and critics, uses a quantile-based ensemble to stabilize value estimation, and selects exploration actors via non-dominated sorting on skill and creativity. The authors prove learning stability and bounded estimation bias, and demonstrate that MAMC achieves faster convergence and stronger performance than several state-of-the-art baselines on MuJoCo continuous-control tasks. The approach offers a scalable way to reduce target-value variance and improve exploration in high-dimensional settings, with code available online. While deterministic policy MAMC excels in many environments, the authors discuss scenarios where stochastic methods may still outperform and suggest future work on adaptive quantile settings and hybrid policy variants.

Abstract

Reinforcement learning has gathered much attention in recent years due to its rapid development and rich applications, especially on control systems and robotics. When tackling real-world applications with reinforcement learning method, the corresponded Markov decision process may have huge discrete or even continuous state/action space. Deep reinforcement learning has been studied for handling these issues through deep learning for years, and one promising branch is the actor-critic architecture. Many past studies leveraged multiple critics to enhance the accuracy of evaluation of a policy for addressing the overestimation and underestimation issues. However, few studies have considered the architecture with multiple actors together with multiple critics. This study proposes a novel multi-actor multi-critic (MAMC) deep deterministic reinforcement learning method. The proposed method has three main features, including selection of actors based on non-dominated sorting for exploration with respect to skill and creativity factors, evaluation for actors and critics using a quantile-based ensemble strategy, and exploiting actors with best skill factor. Theoretical analysis proves the learning stability and bounded estimation bias for the MAMC. The present study examines the performance on a well-known reinforcement learning benchmark MuJoCo. Experimental results show that the proposed framework outperforms state-of-the-art deep deterministic based reinforcement learning methods. Experimental analysis also indicates the proposed components are effective. Empirical analysis further investigates the validity of the proposed method, and shows its benefit on complicated problems. The source code can be found at https://github.com/AndyWu101/MAMC.

Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method

TL;DR

This work introduces Multi-Actor Multi-Critic (MAMC), a deep deterministic RL framework that jointly leverages many actors and critics, uses a quantile-based ensemble to stabilize value estimation, and selects exploration actors via non-dominated sorting on skill and creativity. The authors prove learning stability and bounded estimation bias, and demonstrate that MAMC achieves faster convergence and stronger performance than several state-of-the-art baselines on MuJoCo continuous-control tasks. The approach offers a scalable way to reduce target-value variance and improve exploration in high-dimensional settings, with code available online. While deterministic policy MAMC excels in many environments, the authors discuss scenarios where stochastic methods may still outperform and suggest future work on adaptive quantile settings and hybrid policy variants.

Abstract

Reinforcement learning has gathered much attention in recent years due to its rapid development and rich applications, especially on control systems and robotics. When tackling real-world applications with reinforcement learning method, the corresponded Markov decision process may have huge discrete or even continuous state/action space. Deep reinforcement learning has been studied for handling these issues through deep learning for years, and one promising branch is the actor-critic architecture. Many past studies leveraged multiple critics to enhance the accuracy of evaluation of a policy for addressing the overestimation and underestimation issues. However, few studies have considered the architecture with multiple actors together with multiple critics. This study proposes a novel multi-actor multi-critic (MAMC) deep deterministic reinforcement learning method. The proposed method has three main features, including selection of actors based on non-dominated sorting for exploration with respect to skill and creativity factors, evaluation for actors and critics using a quantile-based ensemble strategy, and exploiting actors with best skill factor. Theoretical analysis proves the learning stability and bounded estimation bias for the MAMC. The present study examines the performance on a well-known reinforcement learning benchmark MuJoCo. Experimental results show that the proposed framework outperforms state-of-the-art deep deterministic based reinforcement learning methods. Experimental analysis also indicates the proposed components are effective. Empirical analysis further investigates the validity of the proposed method, and shows its benefit on complicated problems. The source code can be found at https://github.com/AndyWu101/MAMC.

Paper Structure

This paper contains 20 sections, 4 theorems, 15 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. MAMC
  5. The Overall Procedure
  6. Quantile-based Ensemble Estimation
  7. Actor Evaluation, Exploration, and Exploitation
  8. Theoretical Analysis
  9. Experimental Results
  10. Experimental Settings
  11. Effectiveness
  12. Efficiency
  13. Components Analysis
  14. Sensitivity Analysis
  15. Validity Analysis
  16. ...and 5 more sections

Key Result

Theorem 1

The variance of target values obtained by multiple actors are less than that using a single actor.

Figures (3)

  • Figure 1: Average return against environment steps for TD3-based and SAC-based methods by comparison with the MAMC on the five environments
  • Figure 2: Average return for the best, worst, and skilled (selected) actors in the MAMC in a specific trial on the five test environments
  • Figure 3: Estimated and ground-truth average discounted return against environment steps for the MAMC with different quantile parameters $q$ on HalfCheetah-v5 and Walker2d-v5

Theorems & Definitions (7)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Theorem 2
  • Definition 3
  • Theorem 3
  • Theorem 4