Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditions on Preference Relations that Guarantee the Existence of Optimal Policies

Jonathan Colaço Carr, Prakash Panangaden, Doina Precup

TL;DR

The Direct Preference Process is introduced, a new framework for analyzing LfPF problems in partially-observable, non-Markovian environments and shows that a decision-making problem can have optimal policies -- that are characterized by recursive optimality equations -- even when no reward function can express the learning goal.

Abstract

Learning from Preferential Feedback (LfPF) plays an essential role in training Large Language Models, as well as certain types of interactive learning agents. However, a substantial gap exists between the theory and application of LfPF algorithms. Current results guaranteeing the existence of optimal policies in LfPF problems assume that both the preferences and transition dynamics are determined by a Markov Decision Process. We introduce the Direct Preference Process, a new framework for analyzing LfPF problems in partially-observable, non-Markovian environments. Within this framework, we establish conditions that guarantee the existence of optimal policies by considering the ordinal structure of the preferences. We show that a decision-making problem can have optimal policies -- that are characterized by recursive optimality equations -- even when no reward function can express the learning goal. These findings underline the need to explore preference-based learning strategies which do not assume that preferences are generated by reward.

Conditions on Preference Relations that Guarantee the Existence of Optimal Policies

TL;DR

The Direct Preference Process is introduced, a new framework for analyzing LfPF problems in partially-observable, non-Markovian environments and shows that a decision-making problem can have optimal policies -- that are characterized by recursive optimality equations -- even when no reward function can express the learning goal.

Abstract

Learning from Preferential Feedback (LfPF) plays an essential role in training Large Language Models, as well as certain types of interactive learning agents. However, a substantial gap exists between the theory and application of LfPF algorithms. Current results guaranteeing the existence of optimal policies in LfPF problems assume that both the preferences and transition dynamics are determined by a Markov Decision Process. We introduce the Direct Preference Process, a new framework for analyzing LfPF problems in partially-observable, non-Markovian environments. Within this framework, we establish conditions that guarantee the existence of optimal policies by considering the ordinal structure of the preferences. We show that a decision-making problem can have optimal policies -- that are characterized by recursive optimality equations -- even when no reward function can express the learning goal. These findings underline the need to explore preference-based learning strategies which do not assume that preferences are generated by reward.
Paper Structure (15 sections, 13 theorems, 44 equations, 1 figure)

This paper contains 15 sections, 13 theorems, 44 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Agents and Environments
  5. The Direct Preference Process
  6. Reward-Based Objectives
  7. Conditions for Optimal Policies
  8. Optimal Action Sets
  9. Optimal Feature-based Policies
  10. Embedded Preferences
  11. Connection to Markov Rewards
  12. Conclusion
  13. Missing Proofs
  14. Proof of Lemma \ref{['lemma:stat1']}
  15. Proofs of (F1) and (F2) from Theorem \ref{['thm:markov']}

Key Result

Theorem 10

Let $(\mathcal{O},\mathcal{A},T,e, \preceq)$ be a Direct Preference Process and suppose that a reward function $r:\mathcal{H}\to \mathbb{R}$ expresses $\preceq$. A policy $\pi$ is $\preceq$-optimal if and only if for each attainable history $h_t$,

Figures (1)

  • Figure 1: The Direct Preference Process (DPP) is a new framework for sequential decision-making from preferential feedback. The objectives in a DPP are given by a set of relative preferences over outcomes rather than a reward function. In Section \ref{['sec:optimal-policies']} we show that it is possible for preferences in a DPP to have optimal policies even when no reward function can express the learning goal.

Theorems & Definitions (33)

  • Definition 1
  • Definition 2
  • Definition 3
  • Example 4: Generative Language Models 1
  • Definition 5
  • Definition 6
  • Example 7: Generative Language Models 2
  • Definition 8
  • Definition 9
  • Theorem 10
  • ...and 23 more