Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advances in Preference-based Reinforcement Learning: A Review

Youssef Abdelkareem, Shady Shehata, Fakhri Karray

TL;DR

This survey addresses the core problem of reward engineering in reinforcement learning by embracing preference-based feedback from humans. It presents a unified PbRL framework that encompasses learning directly a policy or learning a surrogate utility function, with both linear and non-linear (notably deep) representations, and discusses theoretical guarantees including regret and finite-sample bounds. The work surveys benchmarking efforts, and highlights practical applications in NLP, especially text summarization, where PbRL demonstrates improved alignment with human judgments. It also candidly discusses limitations—most notably feedback and sample inefficiency—and outlines future directions in safety, theory, and broader real-world applications. Overall, PbRL emerges as a promising direction for scalable, user-involved RL where explicit reward design is difficult, provided advances in efficiency and robustness continue to mature.

Abstract

Reinforcement Learning (RL) algorithms suffer from the dependency on accurately engineered reward functions to properly guide the learning agents to do the required tasks. Preference-based reinforcement learning (PbRL) addresses that by utilizing human preferences as feedback from the experts instead of numeric rewards. Due to its promising advantage over traditional RL, PbRL has gained more focus in recent years with many significant advances. In this survey, we present a unified PbRL framework to include the newly emerging approaches that improve the scalability and efficiency of PbRL. In addition, we give a detailed overview of the theoretical guarantees and benchmarking work done in the field, while presenting its recent applications in complex real-world tasks. Lastly, we go over the limitations of the current approaches and the proposed future research directions.

Advances in Preference-based Reinforcement Learning: A Review

TL;DR

This survey addresses the core problem of reward engineering in reinforcement learning by embracing preference-based feedback from humans. It presents a unified PbRL framework that encompasses learning directly a policy or learning a surrogate utility function, with both linear and non-linear (notably deep) representations, and discusses theoretical guarantees including regret and finite-sample bounds. The work surveys benchmarking efforts, and highlights practical applications in NLP, especially text summarization, where PbRL demonstrates improved alignment with human judgments. It also candidly discusses limitations—most notably feedback and sample inefficiency—and outlines future directions in safety, theory, and broader real-world applications. Overall, PbRL emerges as a promising direction for scalable, user-involved RL where explicit reward design is difficult, provided advances in efficiency and robustness continue to mature.

Abstract

Reinforcement Learning (RL) algorithms suffer from the dependency on accurately engineered reward functions to properly guide the learning agents to do the required tasks. Preference-based reinforcement learning (PbRL) addresses that by utilizing human preferences as feedback from the experts instead of numeric rewards. Due to its promising advantage over traditional RL, PbRL has gained more focus in recent years with many significant advances. In this survey, we present a unified PbRL framework to include the newly emerging approaches that improve the scalability and efficiency of PbRL. In addition, we give a detailed overview of the theoretical guarantees and benchmarking work done in the field, while presenting its recent applications in complex real-world tasks. Lastly, we go over the limitations of the current approaches and the proposed future research directions.
Paper Structure (23 sections, 2 equations, 1 figure)

This paper contains 23 sections, 2 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Design Choices
  4. Preference Type Design Choices
  5. Action Preferences
  6. State Preferences
  7. Trajectory Preferences
  8. Learning Problem Design Choices
  9. Learning a Policy
  10. Learning a utility function
  11. Theoretical Guarantees
  12. Regret Guarantees
  13. Finite-Sample Guarantees
  14. Benchmarking
  15. Application Areas in NLP
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Hierarchical structure of our PbRL framework. Topics in green are surveyed in this work, while the ones in blue are not within our scope. "DNNs" denotes deep neural networks.