Adaptively Private Next-Token Prediction of Large Language Models

TL;DR

Adaptive PMixED (AdaPMixED) is introduced, a private decoding framework based on PMixED that is adaptive to the private and public output distributions evaluated on a given input query that can reduce the privacy loss by 16x while preserving the utility over the original PMixED.

Abstract

As Large Language Models (LLMs) proliferate, developing privacy safeguards for these models is crucial. One popular safeguard involves training LLMs in a differentially private manner. However, such solutions are shown to be computationally expensive and detrimental to the utility of these models. Since LLMs are deployed on the cloud and thus only accessible via an API, a Machine Learning as a Service (MLaaS) provider can protect its downstream data by privatizing the predictions during the decoding process. However, the practicality of such solutions still largely lags behind DP training methods. One recent promising approach, Private Mixing of Ensemble Distributions (PMixED), avoids additive noise by sampling from the output distributions of private LLMs mixed with the output distribution of a public model. Yet, PMixED must satisfy a fixed privacy level for a given number of queries, which is difficult for an analyst to estimate before inference and, hence, does not scale. To this end, we relax the requirements to a more practical setting by introducing Adaptive PMixED (AdaPMixED), a private decoding framework based on PMixED that is adaptive to the private and public output distributions evaluated on a given input query. In this setting, we introduce a noisy screening mechanism that filters out queries with potentially expensive privacy loss, and a data-dependent analysis that exploits the divergence of the private and public output distributions in its privacy loss calculation. Our experimental evaluations demonstrate that our mechanism and analysis can reduce the privacy loss by 16x while preserving the utility over the original PMixED. Furthermore, performing 100K predictions with AdaPMixED still achieves strong utility and a reasonable data-dependent privacy loss of 5.25.

Figures (2)

• Figure 1: A brief overview of AdaPMixED. For each query $\textbf{x}$ received by a user, noisy screening is performed by first mixing each private distribution $p_i(\mathbf{x})$ with the public distribution $p_0(\mathbf{x})$. Then the mixed distribution is privatized with Gaussian noise and is compared with the public distribution $p_0(\mathbf{x})$. If the screening fails, then the next token is sampled from the public distribution $y \sim p_{0}(\mathbf{x})$. Otherwise, private decoding is performed using PMixED. Our privacy analysis tracks the privacy loss $\epsilon(D)$ by analyzing the change of the output distribution when removing one model $p_{-i}(\mathbf{x})$ from decoding with PMixED for every query.
• Figure 2: Ablation study on the privacy-utility tradeoff of privacy parameters, (a) threshold $T$, (b) top-$k$, (c) target leakage $\beta$, (d) Ensemble Size $N$, (e) noisy screening parameters $\sigma, \lambda$, and (f) Renyi Divergence order $\alpha$ for AdaPMixED.

Theorems & Definitions (22)

• Definition 3.1: Approximate DP dwork2014algorithmicfeldman2021individual
• Definition 3.2: Renyi Divergence mironov2017renyi
• Definition 3.3: $(\alpha, \epsilon)$-RDPmironov2017renyifeldman2021individual
• Definition 3.4: Data-Dependent RDP papernot2018scalable
• Theorem 3.1
• Theorem 3.2
• Theorem 4.1
• Theorem 4.2
• Theorem 4.3
• Theorem A.1: Post-Processing mironov2017renyi