Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Bounds for Private and Robust Alignment

Wenqian Weng, Yi He, Xingyu Zhou

TL;DR

This work analyzes private and robust alignment of language models trained with human preferences, addressing both privacy (local differential privacy) and adversarial label corruption in offline and online settings. It shows that standard MLE log loss achieves near-optimal rates under $\varepsilon$-LDP and provides novel uniform-convergence tools that extend to online active exploration, yielding near-optimal rates with online algorithms. It also strengthens guarantees for square-loss based methods in the presence of corruption, obtaining improved bounds in both the corruption-only and joint privacy-corruption regimes, and extends these results to online variants. Collectively, the results advance the theoretical foundations of RLHF-like alignment under privacy and robustness constraints and provide broadly applicable uniform convergence tools for learning theory and statistics.

Abstract

In this paper, we study the private and robust alignment of language models from a theoretical perspective by establishing upper bounds on the suboptimality gap in both offline and online settings. We consider preference labels subject to privacy constraints and/or adversarial corruption, and analyze two distinct interplays between them: privacy-first and corruption-first. For the privacy-only setting, we show that log loss with an MLE-style algorithm achieves near-optimal rates, in contrast to conventional wisdom. For the joint privacy-and-corruption setting, we first demonstrate that existing offline algorithms in fact provide stronger guarantees -- simultaneously in terms of corruption level and privacy parameters -- than previously known, which further yields improved bounds in the corruption-only regime. In addition, we also present the first set of results for private and robust online alignment. Our results are enabled by new uniform convergence guarantees for log loss and square loss under privacy and corruption, which we believe have broad applicability across learning theory and statistics.

Improved Bounds for Private and Robust Alignment

TL;DR

This work analyzes private and robust alignment of language models trained with human preferences, addressing both privacy (local differential privacy) and adversarial label corruption in offline and online settings. It shows that standard MLE log loss achieves near-optimal rates under -LDP and provides novel uniform-convergence tools that extend to online active exploration, yielding near-optimal rates with online algorithms. It also strengthens guarantees for square-loss based methods in the presence of corruption, obtaining improved bounds in both the corruption-only and joint privacy-corruption regimes, and extends these results to online variants. Collectively, the results advance the theoretical foundations of RLHF-like alignment under privacy and robustness constraints and provide broadly applicable uniform convergence tools for learning theory and statistics.

Abstract

In this paper, we study the private and robust alignment of language models from a theoretical perspective by establishing upper bounds on the suboptimality gap in both offline and online settings. We consider preference labels subject to privacy constraints and/or adversarial corruption, and analyze two distinct interplays between them: privacy-first and corruption-first. For the privacy-only setting, we show that log loss with an MLE-style algorithm achieves near-optimal rates, in contrast to conventional wisdom. For the joint privacy-and-corruption setting, we first demonstrate that existing offline algorithms in fact provide stronger guarantees -- simultaneously in terms of corruption level and privacy parameters -- than previously known, which further yields improved bounds in the corruption-only regime. In addition, we also present the first set of results for private and robust online alignment. Our results are enabled by new uniform convergence guarantees for log loss and square loss under privacy and corruption, which we believe have broad applicability across learning theory and statistics.
Paper Structure (25 sections, 19 theorems, 74 equations, 4 algorithms)

This paper contains 25 sections, 19 theorems, 74 equations, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Offline and Online Alignment
  4. Privacy and Robustness in Alignment
  5. Uniform Convergence: Privacy and Corruption
  6. Log Loss under LDP
  7. Square Loss under CTL and LTC
  8. Private Alignment: Log Loss
  9. Offline Setting
  10. Online Setting
  11. Private and Robust Alignment: Square Loss
  12. Offline Setting
  13. Online Setting
  14. Conclusion
  15. Detailed Related Work
  16. ...and 10 more sections

Key Result

Lemma 3.1

Suppose that $\{P_{\theta}(y|x)\}_{\theta \in \Theta} \subseteq (\mathcal{X} \to \Delta(\{-1,1\}))$ is a class of conditional densities parameterized by a finite class $\Theta$. Consider the clean data $(x^{(1)}, y^{(1)}), \ldots, (x^{(T)}, y^{(T)})$ be a sequence of random variables adapted to a fi where $a \lesssim b$ is a shorthand for $a=\mathcal{O}(b)$, $c(\varepsilon) := \frac{e^{\varepsilon

Theorems & Definitions (29)

  • Remark 2.1: Unregularized vs. regularized objective
  • Definition 2.2: Randomized response and $\varepsilon$-LDP warner1965randomized
  • Definition 2.3: $\alpha$-Huber corruption huber1964robust
  • Definition 2.4: $\mathsf{CTL}$ and $\mathsf{LTC}$
  • Remark 2.5
  • Lemma 3.1
  • Remark 3.2
  • Lemma 3.3
  • Remark 3.4
  • Definition 4.3: $L_1$-Concentrability
  • ...and 19 more