Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards User-level Private Reinforcement Learning with Human Feedback

Jiaming Zhang, Mingxi Lei, Meng Ding, Mengdi Li, Zihang Xiang, Difei Xu, Jinhui Xu, Di Wang

TL;DR

This paper tackles user-level privacy in RLHF by introducing AUP-RLHF, a DP-SGD-based framework that enforces $(\varepsilon, \delta)$-user-level label DP while maintaining strong utility. It shows that naive Random Response is suboptimal in the user-level regime and proves a fundamental lower bound on estimation error, then develops AdapUserPriv-SGD within AUP-RLHF to achieve tighter utility with gradient concentration and adaptive data sampling. Theoretical results establish near-optimal estimation error bounds up to polylog factors and a K-wise extension, while experiments on sentiment generation and TL;DR summarization demonstrate superior privacy-utility trade-offs across models and privacy budgets. The approach significantly advances practical RLHF deployments by protecting user preferences without sacrificing performance, enabling privacy-preserving alignment of LLMs with human feedback at scale.

Abstract

Reinforcement Learning with Human Feedback (RLHF) has emerged as an influential technique, enabling the alignment of large language models (LLMs) with human preferences. Despite the promising potential of RLHF, how to protect user preference privacy has become a crucial issue. Most previous work has focused on using differential privacy (DP) to protect the privacy of individual data. However, they have concentrated primarily on item-level privacy protection and have unsatisfactory performance for user-level privacy, which is more common in RLHF. This study proposes a novel framework, AUP-RLHF, which integrates user-level label DP into RLHF. We first show that the classical random response algorithm, which achieves an acceptable performance in item-level privacy, leads to suboptimal utility when in the user-level settings. We then establish a lower bound for the user-level label DP-RLHF and develop the AUP-RLHF algorithm, which guarantees $(\varepsilon, δ)$ user-level privacy and achieves an improved estimation error. Experimental results show that AUP-RLHF outperforms existing baseline methods in sentiment generation and summarization tasks, achieving a better privacy-utility trade-off.

Towards User-level Private Reinforcement Learning with Human Feedback

TL;DR

This paper tackles user-level privacy in RLHF by introducing AUP-RLHF, a DP-SGD-based framework that enforces -user-level label DP while maintaining strong utility. It shows that naive Random Response is suboptimal in the user-level regime and proves a fundamental lower bound on estimation error, then develops AdapUserPriv-SGD within AUP-RLHF to achieve tighter utility with gradient concentration and adaptive data sampling. Theoretical results establish near-optimal estimation error bounds up to polylog factors and a K-wise extension, while experiments on sentiment generation and TL;DR summarization demonstrate superior privacy-utility trade-offs across models and privacy budgets. The approach significantly advances practical RLHF deployments by protecting user preferences without sacrificing performance, enabling privacy-preserving alignment of LLMs with human feedback at scale.

Abstract

Reinforcement Learning with Human Feedback (RLHF) has emerged as an influential technique, enabling the alignment of large language models (LLMs) with human preferences. Despite the promising potential of RLHF, how to protect user preference privacy has become a crucial issue. Most previous work has focused on using differential privacy (DP) to protect the privacy of individual data. However, they have concentrated primarily on item-level privacy protection and have unsatisfactory performance for user-level privacy, which is more common in RLHF. This study proposes a novel framework, AUP-RLHF, which integrates user-level label DP into RLHF. We first show that the classical random response algorithm, which achieves an acceptable performance in item-level privacy, leads to suboptimal utility when in the user-level settings. We then establish a lower bound for the user-level label DP-RLHF and develop the AUP-RLHF algorithm, which guarantees user-level privacy and achieves an improved estimation error. Experimental results show that AUP-RLHF outperforms existing baseline methods in sentiment generation and summarization tasks, achieving a better privacy-utility trade-off.

Paper Structure

This paper contains 30 sections, 15 theorems, 108 equations, 5 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. RLHF
  5. Differential Privacy
  6. Sub-optimality of Random Response
  7. User-level Random Response
  8. Lower Bound of User-level DP-RLHF
  9. Main Method
  10. Experiments
  11. Experimental Setup
  12. Experimental Results
  13. TL;DR Summarization.
  14. Analyzing AUP-RLHF.
  15. Conclusion
  16. ...and 15 more sections

Key Result

Theorem 1

For any $\varepsilon>0$, the private model $\hat{\theta}_{\mathrm{RR}}$ is $\varepsilon$-DP. Moreover, under Assumption ass:1, for any $\alpha>0$, with probability at least $1 - \alpha$, we have where $\gamma = \frac{1}{2 + e^{-2 L B} + e^{2 L B}}$, $\lambda_{\min}(\Sigma_{\mathcal{D}})$ is the minimum eigenvalue of $\Sigma_{\mathcal{D}}$.

Figures (5)

  • Figure 1: IMDb Sentiment Generation (Llama-2-7b, $\epsilon=8$).
  • Figure 2: Win Rate Against the SFT Model for TL;DR Summarization (Gemma-2-2b, $\epsilon=8$).
  • Figure 3: IMDb Sentiment Generation (Llama-2-7b).
  • Figure 4: IMDb Sentiment Generation (Gemma-2-2b).
  • Figure 5: Win Rate Against the SFT Model for TL;DR Summarization (Gemma-2-2b).

Theorems & Definitions (29)

  • Definition 1
  • Definition 2
  • Theorem 1
  • Corollary 1
  • Remark 1
  • Theorem 2: Informal Statement
  • Definition 3
  • Theorem 3
  • Theorem 4
  • Remark 2
  • ...and 19 more