Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Learning from Strategic Human Feedback in LLM Fine-Tuning

Shugang Hao, Lingjie Duan

TL;DR

This work addresses the risk of strategic misreporting in online reinforcement learning from human feedback for LLM fine-tuning. It models the interaction as a dynamic Bayesian game and proposes a non-monetary Online Weighted Aggregation Mechanism that dynamically reweights human labelers based on feedback accuracy, ensuring truthful reporting and a sublinear regret bound of $O(T^{1/2})$. The mechanism outperforms standard average and median aggregation schemes, which suffer from non-vanishing regret. Theoretical guarantees, together with simulations on real-world-like data, demonstrate improved alignment of LLM outputs with diverse human preferences while mitigating long-term manipulation incentives.

Abstract

Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences. However, human labelers are selfish and have diverse preferences. They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences. Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret $\mathcal{O}(T)$ for $T$ time slots. To our best knowledge, we are the first to study online learning mechanisms against strategic human labelers in the LLM fine-tuning process. We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation, ensuring their truthful feedback and sublinear regret $\mathcal{O}(T^{1/2})$. Simulation results demonstrate our mechanism's great advantages over the existing benchmark schemes.

Online Learning from Strategic Human Feedback in LLM Fine-Tuning

TL;DR

This work addresses the risk of strategic misreporting in online reinforcement learning from human feedback for LLM fine-tuning. It models the interaction as a dynamic Bayesian game and proposes a non-monetary Online Weighted Aggregation Mechanism that dynamically reweights human labelers based on feedback accuracy, ensuring truthful reporting and a sublinear regret bound of . The mechanism outperforms standard average and median aggregation schemes, which suffer from non-vanishing regret. Theoretical guarantees, together with simulations on real-world-like data, demonstrate improved alignment of LLM outputs with diverse human preferences while mitigating long-term manipulation incentives.

Abstract

Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences. However, human labelers are selfish and have diverse preferences. They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences. Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret for time slots. To our best knowledge, we are the first to study online learning mechanisms against strategic human labelers in the LLM fine-tuning process. We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation, ensuring their truthful feedback and sublinear regret . Simulation results demonstrate our mechanism's great advantages over the existing benchmark schemes.

Paper Structure

This paper contains 16 sections, 5 theorems, 22 equations, 2 figures.

Table of Contents

  1. Introduction
  2. System Model and Problem Formulation
  3. System Model of LLM Fine-Tuning from Online Human Feedback
  4. Dynamic Bayesian Game Formulation
  5. Two Benchmark Schemes: Average Feedback Aggregation and Median Aggregation
  6. Benchmark 1: Average Feedback Aggregation
  7. Benchmark 2: Median Aggregation Scheme
  8. Online Weighted Aggregation Mechanism: New Design, Analysis, and Simulations
  9. Mechanism Design and Theoretical Analysis
  10. Simulation Results for Verification
  11. Conclusion
  12. Proof of Lemma \ref{['L1']}
  13. Proof of Proposition \ref{['L0']}
  14. Proof of Lemma \ref{['L1-1']}
  15. Proof of Proposition \ref{['L2']}
  16. ...and 1 more sections

Key Result

Lemma 1

The system's regret in e3 under the benchmark 1 of average aggregation is $R_1(T)$=$\mathcal{O}(T)$, leading to a non-vanishing time-average regret $\lim_{T\to\infty}\frac{R_1(T)}{T}> 0$.

Figures (2)

  • Figure 1: Each human labeler's weight $w_i^t$ versus time slot $t$. Here we fix $N=5$, $T=100$, $m_t=50$.
  • Figure 2: Time-average regrets of benchmarks 1, 2, and our mechanism versus the time slot number $T$, respectively. Here we fix $N=100$ and $m_t=50$.

Theorems & Definitions (9)

  • Definition 1: Truthfulness for Human Feedback
  • Definition 2: High Efficiency in Sublinear Regret $R(T)$ in \ref{['e3']}
  • Lemma 1
  • Definition 3: Median Aggregation Scheme
  • Proposition 1
  • Lemma 2
  • Definition 4: Online Weighted Aggregation Mechanism
  • Proposition 2
  • Theorem 1