CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics

TL;DR

The paper addresses the bottlenecks of categorical distributional RL in continuous control by introducing CTD4, a continuous distributional actor-critic that models the return distribution as a Gaussian $\mathcal{N}(\mu,\sigma)$ and uses an ensemble of critics fused via a Kalman filter to mitigate overestimation. The method replaces scalar Q-functions with Gaussian critics, computes a Bellman target $Z_{target} = R + \gamma Z(\mu_k,\sigma_k)$ using Kalman-fused parameters $(\mu_k,\sigma_k)$, and optimizes with KL divergence $D_{KL}$ between current and target distributions. Key contributions include (i) a fully continuous distributional RL framework for continuous actions, (ii) a Kalman fusion strategy that better exploits ensemble information, and (iii) empirical validation on the DeepMind Control Suite showing improved sample efficiency without heavy projection or tuning. This approach offers a scalable, robust alternative to prior CDRL methods and can be integrated into existing TD3-style pipelines to enhance performance in complex, real-world control tasks, with publicly available code and data for reproducibility.

Abstract

Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.

Figures (6)

• Figure 1: Categorical Distribution Computing Problem. The top part represents the sequence of the target distribution being shrunk and shifted by the discount factor $\gamma$ and Reward $R$, respectively. Then, it is projected to the original support through probabilities inversely proportional to the distance from the nearest support. The bottom part describes the current distribution used to calculate the distance error.
• Figure 2: Diagram of the proposed continuous distributional method. The CTD4 architecture network consists of an actor network and $N$ critic networks. Each critic network comprising the ensemble consists of two hidden fully connected layers with 256 nodes, each with ReLU as an activation function. A linear layer is added for each output layer, i.e., the mean $\mu$ and standard deviation $\sigma$. A softplus activation function is also included for $\sigma$ to guarantee positive values. The actor has Tanh as an activation function for the output layer along with two hidden fully connected layers with 256 nodes, each with ReLU.
• Figure 3: The top part represents the sequence of the Z target distribution being shrunk and shifted by the discount factor $\gamma$ and Reward $R$, respectively. However, since these are continuous distributions, no projection steps or estimations of distance from the nearest support are needed. Therefore, the distance between the target and the current distribution can be calculated directly.
• Figure 4: Fusion Method Analysis: Kalman Fusion (Proposed) versus Minimum and Average Value Methods in two environments.
• Figure 5: Ensemble Size Analysis Varying Number of Critics Under Identical Conditions.