ES-C51: Expected Sarsa Based C51 Distributional Reinforcement Learning Algorithm

TL;DR

This paper tackles instability in distributional reinforcement learning when greedy backups obscure nuances across actions with similar means. It introduces ES-C51, an Expected Sarsa based variant of C51 that uses a softmax weighted mixture of next-action distributions for the TD target, with a decaying temperature to recover greediness. Through extensive experiments on Gym classic control and Atari-10, ES-C51 frequently outperforms the baseline QL-C51 while maintaining comparable computational cost, demonstrating improved stability and performance. The work highlights the potential of on-policy distributional methods in deep RL and motivates further research into temperature scheduling and policy churn mitigation.

Abstract

In most value-based reinforcement learning (RL) algorithms, the agent estimates only the expected reward for each action and selects the action with the highest reward. In contrast, Distributional Reinforcement Learning (DRL) estimates the entire probability distribution of possible rewards, providing richer information about uncertainty and variability. C51 is a popular DRL algorithm for discrete action spaces. It uses a Q-learning approach, where the distribution is learned using a greedy Bellman update. However, this can cause problems if multiple actions at a state have similar expected reward but with different distributions, as the algorithm may not learn a stable distribution. This study presents a modified version of C51 (ES-C51) that replaces the greedy Q-learning update with an Expected Sarsa update, which uses a softmax calculation to combine information from all possible actions at a state rather than relying on a single best action. This reduces instability when actions have similar expected rewards and allows the agent to learn higher-performing policies. This approach is evaluated on classic control environments from Gym, and Atari-10 games. For a fair comparison, we modify the standard C51's exploration strategy from e-greedy to softmax, which we refer to as QL-C51 (Q- Learning based C51). The results demonstrate that ES-C51 outperforms QL-C51 across many environments.

Figures (16)

• Figure 1: Example of a stable C51 distribution $Z(s,a_1)$ with low variance.
• Figure 2: Example of an unstable C51 distribution $Z(s,a_2)$ with high variance.
• Figure 3: Performance summary
• Figure 4: Average Performance Improvement by ES-C51
• Figure 5: Performance on Gym Classic Control Environments.