Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On The Fragility of Learned Reward Functions

Lev McKinney, Yawen Duan, David Krueger, Adam Gleave

TL;DR

This paper investigates the fragility of learned reward functions in preference-based reinforcement learning by focusing on relearning failures when freezing a learned reward and training a new agent from scratch. It shows, across tabular and continuous-control tasks, that relearning performance can diverge from the sampler’s success and that this divergence is influenced by reward-model design and the distribution of trajectory data. The results indicate that reward ensembles can mitigate relearning failures by reducing off-distribution reward delusions, while datasets overly concentrated in high-reward regions worsen generalization. The work advocates retraining-based evaluations as a standard diagnostic tool for reward-learning pipelines and highlights practical implications for deploying learned rewards in RL and language-model fine-tuning.

Abstract

Reward functions are notoriously difficult to specify, especially for tasks with complex goals. Reward learning approaches attempt to infer reward functions from human feedback and preferences. Prior works on reward learning have mainly focused on the performance of policies trained alongside the reward function. This practice, however, may fail to detect learned rewards that are not capable of training new policies from scratch and thus do not capture the intended behavior. Our work focuses on demonstrating and studying the causes of these relearning failures in the domain of preference-based reward learning. We demonstrate with experiments in tabular and continuous control environments that the severity of relearning failures can be sensitive to changes in reward model design and the trajectory dataset composition. Based on our findings, we emphasize the need for more retraining-based evaluations in the literature.

On The Fragility of Learned Reward Functions

TL;DR

This paper investigates the fragility of learned reward functions in preference-based reinforcement learning by focusing on relearning failures when freezing a learned reward and training a new agent from scratch. It shows, across tabular and continuous-control tasks, that relearning performance can diverge from the sampler’s success and that this divergence is influenced by reward-model design and the distribution of trajectory data. The results indicate that reward ensembles can mitigate relearning failures by reducing off-distribution reward delusions, while datasets overly concentrated in high-reward regions worsen generalization. The work advocates retraining-based evaluations as a standard diagnostic tool for reward-learning pipelines and highlights practical implications for deploying learned rewards in RL and language-model fine-tuning.

Abstract

Reward functions are notoriously difficult to specify, especially for tasks with complex goals. Reward learning approaches attempt to infer reward functions from human feedback and preferences. Prior works on reward learning have mainly focused on the performance of policies trained alongside the reward function. This practice, however, may fail to detect learned rewards that are not capable of training new policies from scratch and thus do not capture the intended behavior. Our work focuses on demonstrating and studying the causes of these relearning failures in the domain of preference-based reward learning. We demonstrate with experiments in tabular and continuous control environments that the severity of relearning failures can be sensitive to changes in reward model design and the trajectory dataset composition. Based on our findings, we emphasize the need for more retraining-based evaluations in the literature.
Paper Structure (28 sections, 1 equation, 5 figures, 3 tables)

This paper contains 28 sections, 1 equation, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preference-based reward learning
  4. Other reward learning approaches
  5. Reward hacking
  6. Background
  7. Deep RL from Human Preferences
  8. Training and Evaluation Procedure
  9. Reward Learning
  10. Reward Ensembles
  11. Relearning
  12. Experiments
  13. Preference Trajectory Dataset Imbalance and Relearning Failure
  14. Reward Ensembles
  15. Limitations and Discussion
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Anti-correlated sampeler and relearner ground truth returns in HalfCheetah. (a) x-axis represents the number of iterations of each run. See section \ref{['sec:background']} RL budget is the total number of RL timesteps available to the sampler. (b) x-axis represents the number of timesteps during relearning. In plots (a-b), for each RL budget setting, we performed ten runs of reward learning, and for each of these, we ran five relearning evaluations for a total of 50 relearning runs. Solid lines and shaded lines represent the mean and 90% confidence respectively. (c) Scatterplot of average ground truth reward of each segment pair in the example preference datasets with 1M and 8M RL budgets.
  • Figure 2: Ensembles eliminate relearning failures in the stay inside environment. (b) depicts the ground truth reward in the stay inside environment. (c) shows an example individual learned reward and (d) with a five member ensemble. Finally, (e) shows the distribution of max learned-reward across all states. All sub-figures come from the same run which included 20 seeds.
  • Figure 3: Example on policy distribution
  • Figure 4: Increasing the number of time steps of R.L. training does not seem to significantly effect relearning failures.
  • Figure 5: Tiny room environment. The ground-truth reward in the tiny room environment. Note that the reward only depends on the current state.