Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Interactive Reinforcement Learning with Intrinsic Feedback

Benjamin Poole, Minwoo Lee

TL;DR

This tutorial surveys intrinsic feedback as a bridge between reinforcement learning and brain–computer interfaces, elucidating how neural signals can serve as a feedback medium within interactive RL. It formalizes feedback components (type, medium, interpretation, modeling, integration) and reviews foundational LfF approaches (TAMER, COACH, LfP) and their extensions, highlighting how they can be adapted to leverage brain-derived signals like ErrPs. The survey covers motivational justifications, signaling modalities, and proof-of-concept to adaptive strategies for intrinsic feedback, while candidly addressing major challenges such as non-stationary brain signals, credit assignment, and sample efficiency, and it proposes future directions including multi-channel feedback and standardized benchmarking. The work aims to equip BCI and RL researchers with technical grounding to develop robust intrinsic feedback methods and to explore their practical impact on aptness and alignment in real-world human–agent systems.

Abstract

Reinforcement learning (RL) and brain-computer interfaces (BCI) have experienced significant growth over the past decade. With rising interest in human-in-the-loop (HITL), incorporating human input with RL algorithms has given rise to the sub-field of interactive RL. Adjacently, the field of BCI has long been interested in extracting informative brain signals from neural activity for use in human-computer interactions. A key link between these fields lies in the interpretation of neural activity as feedback such that interactive RL approaches can be employed. We denote this new and emerging medium of feedback as intrinsic feedback. Despite intrinsic feedback's ability to be conveyed automatically and even unconsciously, proper exploration surrounding this key link has largely gone unaddressed by both communities. Thus, to help facilitate a deeper understanding and a more effective utilization, we provide a tutorial-style review covering the motivations, approaches, and open problems of intrinsic feedback and its foundational concepts.

Towards Interactive Reinforcement Learning with Intrinsic Feedback

TL;DR

This tutorial surveys intrinsic feedback as a bridge between reinforcement learning and brain–computer interfaces, elucidating how neural signals can serve as a feedback medium within interactive RL. It formalizes feedback components (type, medium, interpretation, modeling, integration) and reviews foundational LfF approaches (TAMER, COACH, LfP) and their extensions, highlighting how they can be adapted to leverage brain-derived signals like ErrPs. The survey covers motivational justifications, signaling modalities, and proof-of-concept to adaptive strategies for intrinsic feedback, while candidly addressing major challenges such as non-stationary brain signals, credit assignment, and sample efficiency, and it proposes future directions including multi-channel feedback and standardized benchmarking. The work aims to equip BCI and RL researchers with technical grounding to develop robust intrinsic feedback methods and to explore their practical impact on aptness and alignment in real-world human–agent systems.

Abstract

Reinforcement learning (RL) and brain-computer interfaces (BCI) have experienced significant growth over the past decade. With rising interest in human-in-the-loop (HITL), incorporating human input with RL algorithms has given rise to the sub-field of interactive RL. Adjacently, the field of BCI has long been interested in extracting informative brain signals from neural activity for use in human-computer interactions. A key link between these fields lies in the interpretation of neural activity as feedback such that interactive RL approaches can be employed. We denote this new and emerging medium of feedback as intrinsic feedback. Despite intrinsic feedback's ability to be conveyed automatically and even unconsciously, proper exploration surrounding this key link has largely gone unaddressed by both communities. Thus, to help facilitate a deeper understanding and a more effective utilization, we provide a tutorial-style review covering the motivations, approaches, and open problems of intrinsic feedback and its foundational concepts.
Paper Structure (37 sections, 37 equations, 10 figures, 4 tables)

This paper contains 37 sections, 37 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning
  4. Brain-Computer Interfaces
  5. Notation
  6. Interactive Reinforcement Learning
  7. Human Input Motivations
  8. Learning from Feedback
  9. Feedback Components
  10. Type
  11. Medium
  12. Interpretation
  13. Model
  14. Integration
  15. Foundational Approaches
  16. ...and 22 more sections

Figures (10)

  • Figure 1: Reinforcement learning setup where agent interacts with the environment at every time step $t$ by observing the current state $S_t$ and performing an action $A_t$ which then produces a reward $R_{t+1}$ and a new state $S_{t+1}$.
  • Figure 2: An example of a simplistic BCI pipeline which includes the stages of acquisition, pre-processing, and classification before being passed to an external device (e.g., computer) for utilization.
  • Figure 3: Time domain
  • Figure 4: Frequency domain
  • Figure 5: Time-Frequency domain
  • ...and 5 more figures