Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning

David Lindner, Mennatallah El-Assady

TL;DR

This work challenges the common assumption that humans can be modeled by simple, unbiased, Boltzmann-like feedback in reinforcement learning. It proposes a framework that treats human feedback as personal, contextual, and dynamic, outlining open research questions and highlighting the need for interdisciplinary collaboration. By categorizing human-AI interactions into instruction, evaluation, and cooperation, and detailing implications for personalized learning, contextualized modeling, and adaptive querying, the paper lays out a roadmap for designing more robust human-in-the-loop RL systems. The authors argue that advancing these models will enable RL to operate effectively in real-world settings where human input is diverse and evolving, potentially sparking new algorithmic directions.

Abstract

Reinforcement learning (RL) commonly assumes access to well-specified reward functions, which many practical applications do not provide. Instead, recently, more work has explored learning what to do from interacting with humans. So far, most of these approaches model humans as being (nosily) rational and, in particular, giving unbiased feedback. We argue that these models are too simplistic and that RL researchers need to develop more realistic human models to design and evaluate their algorithms. In particular, we argue that human models have to be personal, contextual, and dynamic. This paper calls for research from different disciplines to address key questions about how humans provide feedback to AIs and how we can build more robust human-in-the-loop RL systems.

Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning

TL;DR

This work challenges the common assumption that humans can be modeled by simple, unbiased, Boltzmann-like feedback in reinforcement learning. It proposes a framework that treats human feedback as personal, contextual, and dynamic, outlining open research questions and highlighting the need for interdisciplinary collaboration. By categorizing human-AI interactions into instruction, evaluation, and cooperation, and detailing implications for personalized learning, contextualized modeling, and adaptive querying, the paper lays out a roadmap for designing more robust human-in-the-loop RL systems. The authors argue that advancing these models will enable RL to operate effectively in real-world settings where human input is diverse and evolving, potentially sparking new algorithmic directions.

Abstract

Reinforcement learning (RL) commonly assumes access to well-specified reward functions, which many practical applications do not provide. Instead, recently, more work has explored learning what to do from interacting with humans. So far, most of these approaches model humans as being (nosily) rational and, in particular, giving unbiased feedback. We argue that these models are too simplistic and that RL researchers need to develop more realistic human models to design and evaluate their algorithms. In particular, we argue that human models have to be personal, contextual, and dynamic. This paper calls for research from different disciplines to address key questions about how humans provide feedback to AIs and how we can build more robust human-in-the-loop RL systems.
Paper Structure (26 sections, 2 figures)

This paper contains 26 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Position Statement.
  3. Contributions.
  4. Related Work
  5. Collaborative Human-AI Interaction
  6. Instruction.
  7. Evaluation.
  8. Cooperation.
  9. Models of Human Feedback in RL
  10. Challenges in Human Feedback Modelling
  11. Personalized Feedback & Interaction
  12. Contextualized Feedback & Interaction
  13. Adaptive Feedback & Interaction
  14. Implications for RL Applications
  15. Personalized Learning
  16. ...and 11 more sections

Figures (2)

  • Figure 1: We classify a subset of common types of human feedback in RL along three dimensions. Humans can give direct feedback to an RL agent performing a task. The agent can also obtain indirect feedback by observing the human. Feedback can be provided synchronously, i.e., in real-time, or asynchronously, i.e., before or after the agent acts. Finally, feedback is instructive if it tells the agent what to do explicitly; feedback is evaluative if it only tells the agent how good it is doing.
  • Figure 2: Schematic overview of RL with human feedback. In \ref{['sec:modelling_human_feedback']}, we discuss the key components of the interaction. \ref{['sec:rl_implications']} discusses key dimensions for robust interactive learning: Personalized Learning (\ref{['sec:personalized_learning']}), Contextualized Modeling (\ref{['sec:contextualized_learning']}), and Adaptive Querying (\ref{['sec:adaptive_querying']}).