Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tell me why: Training preferences-based RL with human preferences and step-level explanations

Jakob Karalus

TL;DR

The paper tackles the challenge of training agents with human feedback by extending preference-based reinforcement learning to include step-level explanations. It introduces an annotation mechanism and a saliency-based training objective so explanations align with human judgments, yielding a surrogate reward $\hat{r}_\psi$ learned via a Bradley–Terry model and optimized with standard RL methods. The approach combines a dedicated annotation loss $L_{annotation}$ and a structural loss $L_{structural}$ to enforce explanation fidelity and sparsity, and integrates with SAC in a modular, RL-agnostic way. Empirical results on MuJoCo locomotion tasks with a synthetic oracle show faster learning and robust performance under various human irrationalities, suggesting practical benefits for human-in-the-loop learning where richer feedback is feasible.

Abstract

Human-in-the-loop reinforcement learning allows the training of agents through various interfaces, even for non-expert humans. Recently, preference-based methods (PbRL), where the human has to give his preference over two trajectories, increased in popularity since they allow training in domains where more direct feedback is hard to formulate. However, the current PBRL methods have limitations and do not provide humans with an expressive interface for giving feedback. With this work, we propose a new preference-based learning method that provides humans with a more expressive interface to provide their preference over trajectories and a factual explanation (or annotation of why they have this preference). These explanations allow the human to explain what parts of the trajectory are most relevant for the preference. We allow the expression of the explanations over individual trajectory steps. We evaluate our method in various simulations using a simulated human oracle (with realistic restrictions), and our results show that our extended feedback can improve the speed of learning.

Tell me why: Training preferences-based RL with human preferences and step-level explanations

TL;DR

The paper tackles the challenge of training agents with human feedback by extending preference-based reinforcement learning to include step-level explanations. It introduces an annotation mechanism and a saliency-based training objective so explanations align with human judgments, yielding a surrogate reward learned via a Bradley–Terry model and optimized with standard RL methods. The approach combines a dedicated annotation loss and a structural loss to enforce explanation fidelity and sparsity, and integrates with SAC in a modular, RL-agnostic way. Empirical results on MuJoCo locomotion tasks with a synthetic oracle show faster learning and robust performance under various human irrationalities, suggesting practical benefits for human-in-the-loop learning where richer feedback is feasible.

Abstract

Human-in-the-loop reinforcement learning allows the training of agents through various interfaces, even for non-expert humans. Recently, preference-based methods (PbRL), where the human has to give his preference over two trajectories, increased in popularity since they allow training in domains where more direct feedback is hard to formulate. However, the current PBRL methods have limitations and do not provide humans with an expressive interface for giving feedback. With this work, we propose a new preference-based learning method that provides humans with a more expressive interface to provide their preference over trajectories and a factual explanation (or annotation of why they have this preference). These explanations allow the human to explain what parts of the trajectory are most relevant for the preference. We allow the expression of the explanations over individual trajectory steps. We evaluate our method in various simulations using a simulated human oracle (with realistic restrictions), and our results show that our extended feedback can improve the speed of learning.
Paper Structure (14 sections, 7 equations, 3 figures, 1 table)

This paper contains 14 sections, 7 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Preference-based Reinforcement Learning
  5. Training Guided by Saliency-Based Methods
  6. Implementation
  7. Reward Learning with annotated preferences
  8. Structural Loss
  9. Policy Learning
  10. Evaluation
  11. Results
  12. Discussion
  13. Hyperparameters
  14. Compute Power

Figures (3)

  • Figure 1: Overview of our approach. Left: The agent optimizes his policy with respect to the trained proxy reward model. Right: The proxy reward model is trained from human preferences over trajectories and the importance of each timestep in the trajectory. Humans provide both.
  • Figure 2: The mean throughout the training process in each environment. Shaded areas represent the confidence interval. Higher is better.
  • Figure 3: Optimality Gap (how fast a run converges to the ideal score, lower is better) under different human irrationality. The vertical line visualizes the mean optimality gap, and the bars represent the confidence interval.