Diverse Projection Ensembles for Distributional Reinforcement Learning

TL;DR

The paper tackles distributional reinforcement learning (RL) by addressing how projection of return distributions onto representable families biases generalization. It introduces distributional projection ensembles that combine multiple representations (e.g., categorical and quantile) to yield diverse, uncertainty-aware predictions. Theoretical results establish contraction properties for the projection-ensemble operator and a propagation scheme linking distributional TD errors to epistemic uncertainty, enabling optimistic exploration via a learned bonus. The proposed deep RL algorithm PE-DQN uses a two-model ensemble with a mixed quantile/categorical representation, showing improved exploration and learning stability on the bsuite benchmark and VizDoom tasks, with strong performance in hard exploration settings. The work demonstrates that diverse projections improve uncertainty estimation and directed exploration, offering a principled path to robust, sample-efficient distributional RL in complex environments.

Abstract

In contrast to classical reinforcement learning (RL), distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within a set of representable, parametric distributions. Typically, this involves a projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails a strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, we study the combination of several different projections and representations in a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average 1-Wasserstein distance, as a bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and VizDoom and find that diverse projection ensembles lead to significant performance improvements over existing methods on a variety of tasks with the most pronounced gains in directed exploration problems.

Figures (10)

• Figure 1: Toy 1D-regression: Black dots are training data with inputs $x$ and labels $y$. Two models have been trained to predict the distribution $p(y|x)$ using a categorical projection (l.h.s.) and a quantile projection (r.h.s.). We plot contour lines for the $\tau=[0.1,...,0.9]$ quantiles of the predictive distributions over the interval $x\in[-1.5, 1.5]$.
• Figure 2: Illustration of the projection mixture operator with quantile and categorical projections.
• Figure 3: Deep-sea exploration with different statistics. Higher means more exploration. Bars represent medians and interquartile ranges of 30 seeds.
• Figure 4: (a) Summary of bsuite experiments. Wide is better. (b) Median episodic regret for deep sea sizes up to 100. Low is better. Shaded regions are the interquartile range of 10 seeds.
• Figure 5: (a) Visual observation in the VizDoom environment Kempka2016ViZDoom. (b) Mean learning curves in different variations of the MyWayHome VizDoom environment. Shaded regions are $90\%$ Student’s t confidence intervals from 10 seeds.

Theorems & Definitions (16)

• Proposition 1
• Proposition 2
• Theorem 1
• Lemma 1
• Proposition 2
• Proposition 2
• Definition 1
• Definition 2
• Lemma 2
• Theorem 1