Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks

Zihao Li, Xiang Ji, Minshuo Chen, Mengdi Wang

TL;DR

By appropriately selecting the size of a ReLU network, it is shown that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality.

Abstract

A recently popular approach to solving reinforcement learning is with data from human preferences. In fact, human preference data are now used with classic reinforcement learning algorithms such as actor-critic methods, which involve evaluating an intermediate policy over a reward learned from human preference data with distribution shift, known as off-policy evaluation (OPE). Such algorithm includes (i) learning reward function from human preference dataset, and (ii) learning expected cumulative reward of a target policy. Despite the huge empirical success, existing OPE methods with preference data often lack theoretical understanding and rely heavily on heuristics. In this paper, we study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. Specifically, we approach OPE by learning the value function by fitted-Q-evaluation with a deep neural network. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality. Under the assumption of high reward smoothness, our results \textit{almost align with the classical OPE results with observable reward data}. To the best of our knowledge, this is the first result that establishes a \textit{provably efficient} guarantee for off-policy evaluation with RLHF.

Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks

TL;DR

By appropriately selecting the size of a ReLU network, it is shown that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality.

Abstract

A recently popular approach to solving reinforcement learning is with data from human preferences. In fact, human preference data are now used with classic reinforcement learning algorithms such as actor-critic methods, which involve evaluating an intermediate policy over a reward learned from human preference data with distribution shift, known as off-policy evaluation (OPE). Such algorithm includes (i) learning reward function from human preference dataset, and (ii) learning expected cumulative reward of a target policy. Despite the huge empirical success, existing OPE methods with preference data often lack theoretical understanding and rely heavily on heuristics. In this paper, we study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. Specifically, we approach OPE by learning the value function by fitted-Q-evaluation with a deep neural network. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process and obtain a sample-efficient estimator without suffering from the curse of high data ambient dimensionality. Under the assumption of high reward smoothness, our results \textit{almost align with the classical OPE results with observable reward data}. To the best of our knowledge, this is the first result that establishes a \textit{provably efficient} guarantee for off-policy evaluation with RLHF.
Paper Structure (27 sections, 8 theorems, 83 equations, 1 figure, 1 algorithm)

This paper contains 27 sections, 8 theorems, 83 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our results.
  3. Related Work
  4. RLHF for policy learning.
  5. Off-policy evaluation.
  6. Deep neural network in reinforcement learning.
  7. Notation
  8. Preliminaries
  9. Markov Decision Process
  10. Riemannian Manifold
  11. Hölder Functions on Manifolds
  12. ReLU Neural Network Class
  13. Preference-Based Fitted Q-Evaluation
  14. Main Results
  15. Provable benefits of human feedback.
  16. ...and 12 more sections

Key Result

Theorem 4.4

With Assumption ass:rew-smooth, for every $0<\epsilon_0<1$, if the ReLU class $\mathcal{G}_h(\Tilde{R}, \tilde{\tau}, \tilde{L}, \tilde{p}, \tilde{I})$ satisfies then we have for all $h \in [H]$. Here the expectation $\mathbb{E}[\cdot]$ is taken over the data-generating process of ${\mathcal{D}}^{HF}$. Recall that $\eta_h$ is the sampling distribution of $(s,a)$ for the human preference data.

Figures (1)

  • Figure 1: Reward function with low-dim feature.

Theorems & Definitions (18)

  • Definition 2.1: Reach, Definition 2.1 in aamari2019estimating
  • Definition 2.2: Hölder function
  • Theorem 4.4: Reward convergence rate
  • proof
  • Definition 4.5: Restricted $\chi^2$-divergence
  • Theorem 4.6
  • proof
  • proof
  • proof
  • Lemma C.1
  • ...and 8 more