No Free Lunch Theorem for Privacy-Preserving LLM Inference

TL;DR

This work tackles privacy in prompting black-box LLMs by formulating a framework that distorts token embeddings to protect client prompts. It provides a formal No-Free-Lunch theorem bounding the trade-off between privacy leakage and utility loss, showing that a lower leakage budget inevitably incurs higher utility loss. The authors instantiate InferDPT, a DP-based privacy-preserving inference pipeline with a perturbation module and a local extraction module, and validate the theory on CNN/Daily Mail with a GPT-3.5-turbo backend and a Vicuna-7b-4bit local model. The results reveal a clear privacy-utility curve across 24 privacy budgets, confirming the NFL insight and offering guidance for designing practical privacy-preserving LLM systems. The work contributes a rigorous theoretical lens and empirical evidence for balancing privacy and utility in LLM inference, with implications for API-based deployments and privacy-aware prompt design.

Abstract

Individuals and businesses have been significantly benefited by Large Language Models (LLMs) including PaLM, Gemini and ChatGPT in various ways. For example, LLMs enhance productivity, reduce costs, and enable us to focus on more valuable tasks. Furthermore, LLMs possess the capacity to sift through extensive datasets, uncover underlying patterns, and furnish critical insights that propel the frontiers of technology and science. However, LLMs also pose privacy concerns. Users' interactions with LLMs may expose their sensitive personal or company information. A lack of robust privacy safeguards and legal frameworks could permit the unwarranted intrusion or improper handling of individual data, thereby risking infringements of privacy and the theft of personal identities. To ensure privacy, it is essential to minimize the dependency between shared prompts and private information. Various randomization approaches have been proposed to protect prompts' privacy, but they may incur utility loss compared to unprotected LLMs prompting. Therefore, it is essential to evaluate the balance between the risk of privacy leakage and loss of utility when conducting effective protection mechanisms. The current study develops a framework for inferring privacy-protected Large Language Models (LLMs) and lays down a solid theoretical basis for examining the interplay between privacy preservation and utility. The core insight is encapsulated within a theorem that is called as the NFL (abbreviation of the word No-Free-Lunch) Theorem.

Figures (5)

• Figure 1: Privacy Leakage in Prompts. In Case 1, the prompt contains sensitive user information, including: name (John Smith), address (1234 Elm Street, Springfield), and medical condition (high blood pressure). This enables the LLM to identify and extract private data, potentially leading to privacy leakage. Such information risks exposing the user’s identity, location, and health details, which could be misused for identity theft, targeted advertising, or discrimination. In contrast, the prompt in Case 2 excludes sensitive information, significantly reducing the risk of privacy leakage. This comparison highlights how sensitive information in user prompts can be easily identified by LLMs.
• Figure 2: Illustration of the privacy-preserving LLM inference. The LLM inference procedure involves four steps: (1) the client designs a prompt $d$; (2) the client applies a privacy protection mechanism to the original prompt $d$ to obtain a protected prompt $\widetilde{d}$; (3) the client sends $\widetilde{d}$ the server; (4) The LLM takes $\widetilde{d}$ as input and sends the generated response $\widetilde{r}$ back to the client.
• Figure 3: Illustration of randomization protection mechanism. The randomization protection mechanism aims to replace each token in a prompt with a semantically similar token. It typically involves four steps: (1) obtain the embedding $w^{(m)}$ of a given token $d^{(m)}$; (2) add random noise $\delta$ to the embedding $w^{(m)}$ to obtain a distorted embedding $\widetilde{w}^{(m)}$; (3) find adjacent embeddings of $\widetilde{w}^{(m)}$, denoted as $\phi_{d^{(m)}}$; (4) Select an embedding $w$ from $\phi_{d^{(m)}}$ and replace $d^{(m)}$ with token $\widetilde{d}^{(m)}$ corresponding to $w$.
• Figure 4: Privacy Leakage Analysis with Varying Privacy Budget ($\epsilon$). The figure illustrates the relationship between the privacy budget $\epsilon$ and two key metrics: recovery extent and privacy leakage. The top plot displays the recovery extents of the random document $R(\breve P)$ and the perturbed document $R(\widetilde{P})$, while the bottom plot shows the privacy leakage, both as functions of the privacy budget $\epsilon$.
• Figure 5: Privacy and Utility Tradeoff. This figure illustrates the relationship between nine utility loss metrics (vertical axis) and one privacy leakage metric (horizontal axis). Each subplot shows the results for 24 different privacy budgets $\epsilon$, reflecting the corresponding utility loss and privacy leakage. As $\epsilon$ decreases, privacy leakage increases while utility loss decreases. This clearly demonstrates the tradeoff between privacy leakage and utility loss.

Theorems & Definitions (17)

• Definition 3.1: Privacy Leakage
• Definition 3.2: Utility Loss
• Definition 4.1: Optimal Embedding
• Definition 4.2: Near-optimal Embedding
• Definition 4.3: Distortion Extent
• Lemma 4.1: Theoretical Relationship between Recovery Extent and Distortion Extent
• Lemma 4.2: Theoretical Relationship between Privacy Leakage and Total Variation Distance
• Lemma 4.3: Theoretical Relationship between Utility Loss and Total Variation Distance
• Theorem 4.4: No Free Lunch Theorem for Privacy-Preserving LLM Inference
• Lemma Appendix A.1: Theoretical Relationship between Recovery Extent and Distortion Extent