Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Theoretical Framework for Partially Observed Reward-States in RLHF

Chinmaya Kausik, Mirco Mutti, Aldo Pacchiano, Ambuj Tewari

TL;DR

A new history aware version of the Bellman-eluder dimension is defined and a new guarantee for GOLF in the setting, which can be exponentially sharper in illustrative examples, and a naive reduction to cardinal feedback fails to achieve sublinear dueling regret.

Abstract

The growing deployment of reinforcement learning from human feedback (RLHF) calls for a deeper theoretical investigation of its underlying models. The prevalent models of RLHF do not account for neuroscience-backed, partially-observed "internal states" that can affect human feedback, nor do they accommodate intermediate feedback during an interaction. Both of these can be instrumental in speeding up learning and improving alignment. To address these limitations, we model RLHF as reinforcement learning with partially observed reward-states (PORRL). We accommodate two kinds of feedback $-$ cardinal and dueling feedback. We first demonstrate that PORRL subsumes a wide class of RL problems, including traditional RL, RLHF, and reward machines. For cardinal feedback, we present two model-based methods (POR-UCRL, POR-UCBVI). We give both cardinal regret and sample complexity guarantees for the methods, showing that they improve over naive history-summarization. We then discuss the benefits of a model-free method like GOLF with naive history-summarization in settings with recursive internal states and dense intermediate feedback. For this purpose, we define a new history aware version of the Bellman-eluder dimension and give a new guarantee for GOLF in our setting, which can be exponentially sharper in illustrative examples. For dueling feedback, we show that a naive reduction to cardinal feedback fails to achieve sublinear dueling regret. We then present the first explicit reduction that converts guarantees for cardinal regret to dueling regret. In both feedback settings, we show that our models and guarantees generalize and extend existing ones.

A Theoretical Framework for Partially Observed Reward-States in RLHF

TL;DR

A new history aware version of the Bellman-eluder dimension is defined and a new guarantee for GOLF in the setting, which can be exponentially sharper in illustrative examples, and a naive reduction to cardinal feedback fails to achieve sublinear dueling regret.

Abstract

The growing deployment of reinforcement learning from human feedback (RLHF) calls for a deeper theoretical investigation of its underlying models. The prevalent models of RLHF do not account for neuroscience-backed, partially-observed "internal states" that can affect human feedback, nor do they accommodate intermediate feedback during an interaction. Both of these can be instrumental in speeding up learning and improving alignment. To address these limitations, we model RLHF as reinforcement learning with partially observed reward-states (PORRL). We accommodate two kinds of feedback cardinal and dueling feedback. We first demonstrate that PORRL subsumes a wide class of RL problems, including traditional RL, RLHF, and reward machines. For cardinal feedback, we present two model-based methods (POR-UCRL, POR-UCBVI). We give both cardinal regret and sample complexity guarantees for the methods, showing that they improve over naive history-summarization. We then discuss the benefits of a model-free method like GOLF with naive history-summarization in settings with recursive internal states and dense intermediate feedback. For this purpose, we define a new history aware version of the Bellman-eluder dimension and give a new guarantee for GOLF in our setting, which can be exponentially sharper in illustrative examples. For dueling feedback, we show that a naive reduction to cardinal feedback fails to achieve sublinear dueling regret. We then present the first explicit reduction that converts guarantees for cardinal regret to dueling regret. In both feedback settings, we show that our models and guarantees generalize and extend existing ones.
Paper Structure (53 sections, 50 theorems, 171 equations, 1 figure, 8 algorithms)

This paper contains 53 sections, 50 theorems, 171 equations, 1 figure, 8 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Defining RL with Partially-Observed Reward States (PORRL)
  4. PORMDPs
  5. Reinforcement Learning in PORMDPs (PORRL) with Cardinal and Dueling Feedback
  6. Defining the learning objective.
  7. A General Yet Tractable Case
  8. Optimistic Algorithms for Cardinal PORRL
  9. Improving over Naive History-Summarization with Model-Based Methods
  10. Leveraging Recursive Structures Using Model-Free Methods
  11. Dueling to Optimism Reduction
  12. The Naive Reduction Always Fails
  13. Reducing Dueling to Optimistic Cardinal PORRL
  14. Conclusions and Future Work
  15. Lemmas and Discussion
  16. ...and 38 more sections

Key Result

Lemma 1

For any history-dependent policy $\pi$ that selects an action $a_h \sim \pi(\tau[h-1], s_h)$, $V_w(\mathbb{M}, \pi) = V_g(\mathbb{M}, \pi)$ holds for any $\mathbb{M}$.

Figures (1)

  • Figure 1: Illustrating how a human's internal states (represented by emojis) affect their feedback to an agent or LLM. Top: Cardinal or good/bad feedback. Bottom: Dueling or preferential feedback. In line with Definition \ref{['def:pormdp']}, $u_h \in {\cal U}$ are represented by the emojis, $p=2$ and ${\cal H}_p = \{2, 4\}$ in both cases.

Theorems & Definitions (91)

  • Definition 1
  • Definition 2
  • Lemma 1: Replacing $w$ with $g$
  • Remark 1
  • Definition 3
  • Remark 2
  • Theorem 1: POR-UCRL Regret
  • Remark 3: Comparison to past results
  • Remark 4: General function approximation for ${\cal P}$
  • Definition 4
  • ...and 81 more