Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning

Yinglun Xu, David Zhu, Rohan Gumaste, Gagandeep Singh

TL;DR

This work proposes a general framework for transforming preference feedback to scalar rewards via binary reward labeling (BRL), and then any reward-based offline RL algorithms can be applied to the dataset with the reward labels.

Abstract

Offline reinforcement learning has become one of the most practical RL settings. However, most existing works on offline RL focus on the standard setting with scalar reward feedback. It remains unknown how to universally transfer the existing rich understanding of offline RL from the reward-based to the preference-based setting. In this work, we propose a general framework to bridge this gap. Our key insight is transforming preference feedback to scalar rewards via binary reward labeling (BRL), and then any reward-based offline RL algorithms can be applied to the dataset with the reward labels. The information loss during the feedback signal transition is minimized with binary reward labeling in the practical learning scenarios. We theoretically show the connection between several recent PBRL techniques and our framework combined with specific offline RL algorithms. By combining reward labeling with different algorithms, our framework can lead to new and potentially more efficient offline PBRL algorithms. We empirically test our framework on preference datasets based on the standard D4RL benchmark. When combined with a variety of efficient reward-based offline RL algorithms, the learning result achieved under our framework is comparable to training the same algorithm on the dataset with actual rewards in many cases and better than the recent PBRL baselines in most cases.

Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning

TL;DR

This work proposes a general framework for transforming preference feedback to scalar rewards via binary reward labeling (BRL), and then any reward-based offline RL algorithms can be applied to the dataset with the reward labels.

Abstract

Offline reinforcement learning has become one of the most practical RL settings. However, most existing works on offline RL focus on the standard setting with scalar reward feedback. It remains unknown how to universally transfer the existing rich understanding of offline RL from the reward-based to the preference-based setting. In this work, we propose a general framework to bridge this gap. Our key insight is transforming preference feedback to scalar rewards via binary reward labeling (BRL), and then any reward-based offline RL algorithms can be applied to the dataset with the reward labels. The information loss during the feedback signal transition is minimized with binary reward labeling in the practical learning scenarios. We theoretically show the connection between several recent PBRL techniques and our framework combined with specific offline RL algorithms. By combining reward labeling with different algorithms, our framework can lead to new and potentially more efficient offline PBRL algorithms. We empirically test our framework on preference datasets based on the standard D4RL benchmark. When combined with a variety of efficient reward-based offline RL algorithms, the learning result achieved under our framework is comparable to training the same algorithm on the dataset with actual rewards in many cases and better than the recent PBRL baselines in most cases.
Paper Structure (23 sections, 3 theorems, 13 equations, 6 figures, 2 tables, 1 algorithm)

This paper contains 23 sections, 3 theorems, 13 equations, 6 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Offline Reward-Based Reinforcement Learning
  4. Offline Preference-Based Reinforcement Learning
  5. Preliminaries
  6. Reinforcement Learning
  7. Preference Based Reinforcement Learning (PBRL)
  8. Offline PBRL Setting
  9. Bridging Preference and Reward-Based Offline Reinforcement Learning
  10. Optimal Reward Labeling
  11. Theoretical Analysis
  12. Experiments
  13. Experiments Setup
  14. Learning Efficiency Evaluation without Trajectory Overlap
  15. Overlapped Trajectories
  16. ...and 8 more sections

Key Result

Lemma 4.2

Consider a preference dataset $D=\{(\tau_1^i \succ \tau_2^i)\}, i \in [N]$. If each state-action pair is unique in the dataset, the optimal reward labels are:

Figures (6)

  • Figure 1: Illustration of the binary reward labeling (BRL) framework. Given a dataset consisting of preference feedback as input, the BRL framework labels the dataset with the rewards that best explain the preference signals from the dataset. With a dataset of reward labels, the learning agent can train on the dataset with any efficient offline RL algorithms such as IQL kostrikov2021offline or CQL kumar2020conservative and enjoy high learning efficiency.
  • Figure 2: Training log of learning with a method on datasets with overlapped trajectories. The percentage is the portion of trajectories clips that are compared for multiple times compared to all clips. The multiplier is the number of times of multiple comparison. To understand the degree of overlap, in the case of $20\% \times 4$, $80\%$ of the trajectories pairs have a trajectory clip that is compared for multiple times.
  • Figure 3: Training log of learning with a method on datasets where $10$ preference labels are given to each trajectory pair.
  • Figure 4: Training log of learning with a method on datasets of different sizes. The percentage in the legends represents the ratio between the number of preference labels in the corresponding dataset and that of the largest dataset.
  • Figure 5: Comparison between the learning efficiency of BRL combined with different standard offline RL algorithms.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Definition 4.1: Optimal reward label
  • Lemma 4.2
  • Definition 4.3
  • Definition 4.4
  • Theorem 4.5
  • Definition A.1
  • Definition A.2
  • Theorem A.3