Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

James Flemings, Haosheng Gan, Hongyi Li, Meisam Razaviyayn, Murali Annavaram

TL;DR

This work tackles privacy leakage in in-context learning by enabling differentially private generation without added noise. It introduces DPS-MOZO, a decoding framework that mixes subsampled one-shot demonstrations with a zero-shot distribution via adaptive lambda parameters to bound Renyi divergence and achieve DP. By operating in online (On) or offline (Off) modes, DPS-MOZO provides strong privacy guarantees (e.g., $\epsilon=2$) with minimal utility loss on multiple generation benchmarks, often surpassing existing DP-ICL baselines. The approach leverages privacy amplification by subsampling and the post-processing property of DP, offering a practical path toward privacy-preserving LLM-based ICL in real-world deployment.

Abstract

In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose $\texttt{dps-mozo}$, Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show $\texttt{dps-mozo}$ can achieve a strong privacy guarantee, $ε=2$, with minimal utility degradation compared to non-private few-shot learning, $\textbf{0.3}$% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.

Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

TL;DR

This work tackles privacy leakage in in-context learning by enabling differentially private generation without added noise. It introduces DPS-MOZO, a decoding framework that mixes subsampled one-shot demonstrations with a zero-shot distribution via adaptive lambda parameters to bound Renyi divergence and achieve DP. By operating in online (On) or offline (Off) modes, DPS-MOZO provides strong privacy guarantees (e.g., ) with minimal utility loss on multiple generation benchmarks, often surpassing existing DP-ICL baselines. The approach leverages privacy amplification by subsampling and the post-processing property of DP, offering a practical path toward privacy-preserving LLM-based ICL in real-world deployment.

Abstract

In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose , Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show can achieve a strong privacy guarantee, , with minimal utility degradation compared to non-private few-shot learning, % ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.

Paper Structure

This paper contains 22 sections, 7 theorems, 12 equations, 7 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Background
  4. Methodology
  5. DPS-MOZO
  6. Applying DPS-MOZO for Privacy-preserving ICL
  7. Experiments
  8. Experimental Setup
  9. Main Results
  10. Insights Behind the Performance of DPS-MOZO
  11. Ablation Studies
  12. Empirical Privacy Evaluation
  13. Discussion and Conclusion
  14. Properties of RDP
  15. Detailed Privacy Analysis of DPS-MOZO
  16. ...and 7 more sections

Key Result

Theorem 1.1

Let $A: \mathcal{D} \rightarrow \mathcal{R}$ be $(\alpha, \epsilon(\alpha))$-RDP, and let $F: \mathcal{R} \rightarrow \mathcal{Z}$ be an arbitrary randomized mapping. Then $F \circ M$ is $(\alpha, \epsilon(\alpha))$-RDP.

Figures (7)

  • Figure 1: Problem setup for privacy-preserving in-context learning.
  • Figure 2: (a) A high-level overview of DPS-MOZO broken down into four phases: (1) Subsample Demonstrations. First, $n_{\text{shots}}$ demonstrations are randomly selected without replacement from a private dataset, then one-shot prompts containing a demonstration, the input query, and the current output are generated. Also, a zero-shot prompt containing only the input query and the current output is generated. (2) Inference. An LLM performs inference on each prompt to generate $n_{\text{shots}}+1$ output distributions. (3) Mixing. Each one-shot output is mixed with the zero-shot output such that the resulting mixed output has a bounded divergence from the zero-shot output. (4) Decode. Lastly, the next token is sampled from the product of the mixed one-shot output distributions. (b) Two applications of DPS-MOZO on an ICL pipeline. The first, DPS-MOZO+On, applies DPS-MOZO during the online phase to answer input queries. The second, DPS-MOZO+Off, applies DPS-MOZO during the offline phase with public inputs to generate DP outputs, both of which are to be used in the online phase.
  • Figure 3: Tracking $\lambda^{(t)}$ from DPS-MOZO+On, the average value across all test queries of the smallest $\lambda_i$ (y-axis) at the $t$-th generation (x-axis).
  • Figure 4: Ablation studies on dialogue summarization using the SAMSum dataset.
  • Figure 5: Example of one-shot and zero-shot prompts used for the E2E dataset where red text is an example input-output pair $(\mathbf{x}_i, \mathbf{y}_i)$ and blue text is the input query $\boldsymbol{x}$.
  • ...and 2 more figures

Theorems & Definitions (10)

  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Theorem 1.1: Post-Processing mironov2017renyi
  • Theorem 1.2: Composition mironov2017renyi
  • Theorem 1.3: Conversion from RDP to Approximate DP balle2020hypothesis
  • Theorem 1.4: RDP for Subsampled Mechanisms wang2019subsampled.
  • Theorem 2.1: Triangle-like inequality, lemma 33.7 from steinke2022composition
  • Theorem 2.2: Additivity, Theorem 28 from van2014renyi
  • Theorem 2.3