Distorted Distributional Policy Evaluation for Offline Reinforcement Learning
Ryo Iwaki, Takayuki Osogami
TL;DR
The paper tackles the challenge that offline distributional reinforcement learning suffers from uniformly pessimistic estimates of return quantiles, which can hinder generalization. It introduces Distorted Distributional Evaluation (DDE) by applying a quantile distortion $\mathcal{Q}_{\phi}$, driven by ensemble-based uncertainty $\phi(s,a,\tau)$, to realize non-uniform tail pessimism. The authors prove contraction properties of the distortion-augmented operator and derive fixed-point bounds that relate distorted and undistorted value functions, supporting tail-focused pessimism. Empirically, they develop Distorted Distributional Actor Critic (DDAC) and show it achieves more stable learning and higher final performance than uniform-pessimism baselines like CODAC on offline benchmarks, highlighting the approach's practical potential for risk-aware offline RL and AutoRL pipelines.
Abstract
While Distributional Reinforcement Learning (DRL) methods have demonstrated strong performance in online settings, its success in offline scenarios remains limited. We hypothesize that a key limitation of existing offline DRL methods lies in their approach to uniformly underestimate return quantiles. This uniform pessimism can lead to overly conservative value estimates, ultimately hindering generalization and performance. To address this, we introduce a novel concept called quantile distortion, which enables non-uniform pessimism by adjusting the degree of conservatism based on the availability of supporting data. Our approach is grounded in theoretical analysis and empirically validated, demonstrating improved performance over uniform pessimism.