Discerning Temporal Difference Learning
Jianfei Ma
TL;DR
This work tackles credit assignment in temporal-difference learning under state visitation imbalance and reward noise by introducing discerning Temporal Difference Learning (DTD), which integrates a nonnegative emphasis function to reweight TD error propagation and a discerning lambda-return to modulate multi-step returns. The authors establish convergence of DTD for a broad class of emphasis functions and discuss extensions to deep RL via a discerning advantage estimator and connections to prioritized sampling. Empirically, DTD demonstrates improved value estimation and faster learning across varied tasks, with adaptive emphasis outperforming baselines and showing robustness to noise and representation quality. Overall, the framework offers flexible, emphasis-aware credit assignment that can be integrated with DRL techniques to enhance stability and efficiency in learning.
Abstract
Temporal difference learning (TD) is a foundational concept in reinforcement learning (RL), aimed at efficiently assessing a policy's value function. TD($λ$), a potent variant, incorporates a memory trace to distribute the prediction error into the historical context. However, this approach often neglects the significance of historical states and the relative importance of propagating the TD error, influenced by challenges such as visitation imbalance or outcome noise. To address this, we propose a novel TD algorithm named discerning TD learning (DTD), which allows flexible emphasis functions$-$predetermined or adapted during training$-$to allocate efforts effectively across states. We establish the convergence properties of our method within a specific class of emphasis functions and showcase its promising potential for adaptation to deep RL contexts. Empirical results underscore that employing a judicious emphasis function not only improves value estimation but also expedites learning across diverse scenarios.