Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards

Xi Chen, Zhihui Zhu, Andrew Perrault

TL;DR

This work tackles reinforcement learning under perturbed rewards, where the observed rewards may be noisy, corrupted, or adversarially modified. It introduces the Distributional Reward Critic (DRC) and its Generalized extension (GDRC), reframing reward estimation as distributional classification over $n_r$ bins and employing ordinal cross-entropy to preserve reward ordering, even under unknown perturbations modeled by a Generalized Confusion Matrix (GCM). GDRC further extends DRC by training an ensemble across discretizations to infer the perturbation structure online and by adapting the reward range with a moving window, enabling robust learning across a broad class of perturbations. Empirically, DRC and GDRC outperform state-of-the-art baselines on Mujoco and discrete tasks under both GCM and continuous perturbations, with GDRC achieving wins or ties in 44/48 settings, demonstrating strong practical applicability for RL in perturbed reward environments. The methods require no prior knowledge about the perturbations and leverage a distributional perspective to denoise rewards and adapt to shifting optimal policies, offering a broadly applicable framework for robust RL.

Abstract

The reward signal plays a central role in defining the desired behaviors of agents in reinforcement learning (RL). Rewards collected from realistic environments could be perturbed, corrupted, or noisy due to an adversary, sensor error, or because they come from subjective human feedback. Thus, it is important to construct agents that can learn under such rewards. Existing methodologies for this problem make strong assumptions, including that the perturbation is known in advance, clean rewards are accessible, or that the perturbation preserves the optimal policy. We study a new, more general, class of unknown perturbations, and introduce a distributional reward critic framework for estimating reward distributions and perturbations during training. Our proposed methods are compatible with any RL algorithm. Despite their increased generality, we show that they achieve comparable or better rewards than existing methods in a variety of environments, including those with clean rewards. Under the challenging and generalized perturbations we study, we win/tie the highest return in 44/48 tested settings (compared to 11/48 for the best baseline). Our results broaden and deepen our ability to perform RL in reward-perturbed environments.

The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards

TL;DR

This work tackles reinforcement learning under perturbed rewards, where the observed rewards may be noisy, corrupted, or adversarially modified. It introduces the Distributional Reward Critic (DRC) and its Generalized extension (GDRC), reframing reward estimation as distributional classification over bins and employing ordinal cross-entropy to preserve reward ordering, even under unknown perturbations modeled by a Generalized Confusion Matrix (GCM). GDRC further extends DRC by training an ensemble across discretizations to infer the perturbation structure online and by adapting the reward range with a moving window, enabling robust learning across a broad class of perturbations. Empirically, DRC and GDRC outperform state-of-the-art baselines on Mujoco and discrete tasks under both GCM and continuous perturbations, with GDRC achieving wins or ties in 44/48 settings, demonstrating strong practical applicability for RL in perturbed reward environments. The methods require no prior knowledge about the perturbations and leverage a distributional perspective to denoise rewards and adapt to shifting optimal policies, offering a broadly applicable framework for robust RL.

Abstract

The reward signal plays a central role in defining the desired behaviors of agents in reinforcement learning (RL). Rewards collected from realistic environments could be perturbed, corrupted, or noisy due to an adversary, sensor error, or because they come from subjective human feedback. Thus, it is important to construct agents that can learn under such rewards. Existing methodologies for this problem make strong assumptions, including that the perturbation is known in advance, clean rewards are accessible, or that the perturbation preserves the optimal policy. We study a new, more general, class of unknown perturbations, and introduce a distributional reward critic framework for estimating reward distributions and perturbations during training. Our proposed methods are compatible with any RL algorithm. Despite their increased generality, we show that they achieve comparable or better rewards than existing methods in a variety of environments, including those with clean rewards. Under the challenging and generalized perturbations we study, we win/tie the highest return in 44/48 tested settings (compared to 11/48 for the best baseline). Our results broaden and deepen our ability to perform RL in reward-perturbed environments.
Paper Structure (42 sections, 2 theorems, 1 equation, 15 figures, 9 tables)

This paper contains 42 sections, 2 theorems, 1 equation, 15 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The Distributional Reward Critic (DRC) Framework
  3. General Distributional Reward Critic (GDRC)
  4. Experimental Evaluation
  5. Related Work
  6. Reward Prediction By The Reward Estimation (RE) Method
  7. Perturbation Modeling By The Surrogate Reward (SR) Method
  8. Distributional RL
  9. Problem Statement
  10. Perturbed Reward MDP
  11. Generalized Confusion Matrix (GCM) Perturbations
  12. The Distributional Reward Critic Architecture
  13. Distributional Reward Critic (DRC)
  14. General Distributional Reward Critic (GDRC)
  15. Impact Of $n_o$ On Reconstruction Error For DRC
  16. ...and 27 more sections

Key Result

Proposition 1

Consider continuous perturbations that for each reward $r\in[r_{\min},r_{\max})$, it can be perturbed to $\bar{r} \in[r_{\min},r_{\max})$. Our GCM represents $\bar{r}$ with $\tilde{r}$ that satisfies $|\tilde{r} - \bar{r}|\le \frac{r_{\max} - r_{\min}}{n_r}$.

Figures (15)

  • Figure 1: An example of GCM perturbation.
  • Figure 2: The pipeline of the whole process and the pseudocode of distributional reward methods.
  • Figure 3: Illustration of reconstruction error and cross entropy as $n_o$ varies in simulation environments where $n_r=5$.
  • Figure 4: Illustration of cross-entropy for different $n_o$.
  • Figure 5: The results of Mujoco environments under GCM perturbations. Solid line methods, which are of greater interest, can be applied without any prior information. DRC and GDRC are our methods. The $x$-axis represents perturbation ratio $\omega$, and the $y$-axis represents performance.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Proposition 1
  • Proposition B.1
  • proof