Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Off-Policy Evaluation from Logged Human Feedback

Aniruddha Bhargava, Lalit Jain, Branislav Kveton, Ge Liu, Subhojyoti Mukherjee

TL;DR

The paper tackles evaluating a new policy using only previously logged human feedback on ranked responses, addressing the cost of collecting fresh feedback in RLHF/DPO pipelines. It introduces a Plackett-Luce based feedback model and develops both model-based (Direct Method) and model-free (IPS, DR) estimators, complemented by SetIPS and SetDR variance-reducing variants and a KL-regularized optimization framework. The authors provide unbiasedness analyses, demonstrate variance reductions, and validate the approach on synthetic data and a real Nectar LLM dataset, including a policy-optimization demonstration. Overall, the work enables reuse of existing human feedback to both evaluate and improve policy alignment, reducing the need for costly new human studies in large-scale language model workflows.

Abstract

Learning from human feedback has been central to recent advances in artificial intelligence and machine learning. Since the collection of human feedback is costly, a natural question to ask is if the new feedback always needs to collected. Or could we evaluate a new model with the human feedback on responses of another model? This motivates us to study off-policy evaluation from logged human feedback. We formalize the problem, propose both model-based and model-free estimators for policy values, and show how to optimize them. We analyze unbiasedness of our estimators and evaluate them empirically. Our estimators can predict the absolute values of evaluated policies, rank them, and be optimized.

Off-Policy Evaluation from Logged Human Feedback

TL;DR

The paper tackles evaluating a new policy using only previously logged human feedback on ranked responses, addressing the cost of collecting fresh feedback in RLHF/DPO pipelines. It introduces a Plackett-Luce based feedback model and develops both model-based (Direct Method) and model-free (IPS, DR) estimators, complemented by SetIPS and SetDR variance-reducing variants and a KL-regularized optimization framework. The authors provide unbiasedness analyses, demonstrate variance reductions, and validate the approach on synthetic data and a real Nectar LLM dataset, including a policy-optimization demonstration. Overall, the work enables reuse of existing human feedback to both evaluate and improve policy alignment, reducing the need for costly new human studies in large-scale language model workflows.

Abstract

Learning from human feedback has been central to recent advances in artificial intelligence and machine learning. Since the collection of human feedback is costly, a natural question to ask is if the new feedback always needs to collected. Or could we evaluate a new model with the human feedback on responses of another model? This motivates us to study off-policy evaluation from logged human feedback. We formalize the problem, propose both model-based and model-free estimators for policy values, and show how to optimize them. We analyze unbiasedness of our estimators and evaluate them empirically. Our estimators can predict the absolute values of evaluated policies, rank them, and be optimized.
Paper Structure (16 sections, 6 theorems, 39 equations, 4 figures)

This paper contains 16 sections, 6 theorems, 39 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Setting
  3. Off-Policy Evaluation
  4. Direct Method
  5. Propensity-Based Methods
  6. Advanced Propensity-Based Methods
  7. Off-Policy Optimization
  8. Experiments
  9. Absolute Error
  10. Relative Error
  11. Large Language Models
  12. Policy Optimization
  13. Related Work
  14. Conclusions
  15. Technical Lemmas
  16. ...and 1 more sections

Key Result

Lemma 1

Let $\hat{V}(\pi)$ be defined as in eq:counter. Then $\mathbb{E} \left[\hat{V}(\pi)\right] = V(\pi)$.

Figures (4)

  • Figure 1: Absolute error of estimated policy values as we vary $K$, $n$, and $\sigma_\phi$.
  • Figure 2: Ranking error of estimated policy values as we vary $K$, $n$, and $\sigma_\phi$.
  • Figure 3: Evaluation of large language model policies on Nectar dataset by the absolute error. The parameter $\alpha \in [0, 1]$ interpolates between Phi3 and Llama3 policies.
  • Figure 4: Policy optimization of our estimators, together with RLHF and DPO. The plots are smoothed out by averaging over a window of $50$ steps.

Theorems & Definitions (12)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • proof
  • ...and 2 more