Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Private is Your Attention? Bridging Privacy with In-Context Learning

Soham Bonnerjee, Zhen Wei, Yeon, Anna Asch, Sagnik Nandy, Promit Ghosal

TL;DR

This paper addresses privacy preservation during pretraining for in-context learning (ICL) by introducing NoisyHead, a differentially-private pretraining method for linear attention heads that enables ICL in linear regression. It provides a rigorous analysis of the privacy–utility trade-off, revealing distinct regimes for low- and high-dimensional settings and highlighting the importance of early stopping to balance gradient-noise and optimization error. The work also proves robustness to adversarial perturbations in training prompts and validates theoretical predictions with extensive numerical experiments across regimes, including an over-parameterized phase transition. Collectively, these results offer a principled pathway for privacy-preserving ICL in transformer-based architectures and quantify the practical costs and safeguards of private pretraining.

Abstract

In-context learning (ICL)-the ability of transformer-based models to perform new tasks from examples provided at inference time-has emerged as a hallmark of modern language models. While recent works have investigated the mechanisms underlying ICL, its feasibility under formal privacy constraints remains largely unexplored. In this paper, we propose a differentially private pretraining algorithm for linear attention heads and present the first theoretical analysis of the privacy-accuracy trade-off for ICL in linear regression. Our results characterize the fundamental tension between optimization and privacy-induced noise, formally capturing behaviors observed in private training via iterative methods. Additionally, we show that our method is robust to adversarial perturbations of training prompts, unlike standard ridge regression. All theoretical findings are supported by extensive simulations across diverse settings.

How Private is Your Attention? Bridging Privacy with In-Context Learning

TL;DR

This paper addresses privacy preservation during pretraining for in-context learning (ICL) by introducing NoisyHead, a differentially-private pretraining method for linear attention heads that enables ICL in linear regression. It provides a rigorous analysis of the privacy–utility trade-off, revealing distinct regimes for low- and high-dimensional settings and highlighting the importance of early stopping to balance gradient-noise and optimization error. The work also proves robustness to adversarial perturbations in training prompts and validates theoretical predictions with extensive numerical experiments across regimes, including an over-parameterized phase transition. Collectively, these results offer a principled pathway for privacy-preserving ICL in transformer-based architectures and quantify the practical costs and safeguards of private pretraining.

Abstract

In-context learning (ICL)-the ability of transformer-based models to perform new tasks from examples provided at inference time-has emerged as a hallmark of modern language models. While recent works have investigated the mechanisms underlying ICL, its feasibility under formal privacy constraints remains largely unexplored. In this paper, we propose a differentially private pretraining algorithm for linear attention heads and present the first theoretical analysis of the privacy-accuracy trade-off for ICL in linear regression. Our results characterize the fundamental tension between optimization and privacy-induced noise, formally capturing behaviors observed in private training via iterative methods. Additionally, we show that our method is robust to adversarial perturbations of training prompts, unlike standard ridge regression. All theoretical findings are supported by extensive simulations across diverse settings.

Paper Structure

This paper contains 27 sections, 9 theorems, 53 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Main Results
  3. Related literature and notations
  4. Notation
  5. Problem Formulation
  6. Differentially Private Pretraining
  7. Cost of In-Context Differential Privacy
  8. Robustness properties of NoisyHead
  9. Numerical experiments
  10. Effect of privacy on prediction risk
  11. Effect of early stopping in over-parametrized setting
  12. Robustness of NoisyHead
  13. Conclusion
  14. Theoretical Details
  15. Choice of parameters
  16. ...and 12 more sections

Key Result

Theorem 1.1

In the low dimensional regime, when $L$ and $\sqrt{N}$ are asymptotically of same order and $D = O(1)$, the cost of privacy satisfies In the high dimensional regime, when $N/D^2 = O(1)$ and $L / D = O(1)$, the cost of privacy scales as up to $\mathrm{polylog}$ factors.

Figures (4)

  • Figure 1: Excess risk of NoisyHead for the low-dimensional set-up with $D=5$.
  • Figure 2: Interplay between the cost of descent and the cost of privacy in the overparameterized setting with $N=1000$ and $\varepsilon=0.8$.
  • Figure 3: Comparison of prediction error under adversarial perturbations for different values of $c$. Left: $c = 2$; Right: $c = 4$. The differentially private estimator (NoisyHead) consistently outperforms the ridge estimator ($\Gamma^\star$) as the perturbation magnitude $\alpha = cN^p$ increases.
  • Figure 4: Excess risk of NoisyHead as a function of training set size $N$ for different values of the privacy parameter $\varepsilon$ with $D=\lfloor\sqrt{N}\rfloor$.

Theorems & Definitions (13)

  • Theorem 1.1: Informal
  • Definition 3.1
  • Theorem 3.2
  • Theorem 4.1
  • Remark 4.1
  • Theorem 4.2
  • Remark 4.2
  • Proposition 4.1
  • Theorem 5.1
  • Remark 5.1
  • ...and 3 more