Privacy Preserving Prompt Engineering: A Survey

TL;DR

This survey addresses privacy risks arising in prompting and in-context learning (ICL) for large language models, focusing on protecting sensitive information embedded in prompts and demonstration examples. It provides a taxonomy of privacy-preserving prompting methods categorized as non-DP, local DP, global DP, and other scenarios, and links each approach to its privacy targets, discussing mechanisms from sanitization to cryptography and DP-based frameworks (e.g., DP-SGD, PATE). The resource compendium covers datasets and tools across NLP, tabular, and image domains, and highlights limitations in computational efficiency, semantic fidelity, and trustworthiness. It outlines future directions for computationally efficient private prompting, improved privacy-utility tradeoffs, broader modality extension, and standardized benchmarks, aiming to guide researchers and practitioners toward practical privacy-preserving prompting solutions. Budgets such as $(\varepsilon, \delta)$-DP and techniques like DP-SGD and PATE are central to many Global-DP approaches analyzed in the survey.

Abstract

Pre-trained language models (PLMs) have demonstrated significant proficiency in solving a wide range of general natural language processing (NLP) tasks. Researchers have observed a direct correlation between the performance of these models and their sizes. As a result, the sizes of these models have notably expanded in recent years, persuading researchers to adopt the term large language models (LLMs) to characterize the larger-sized PLMs. The size expansion comes with a distinct capability called in-context learning (ICL), which represents a special form of prompting and allows the models to be utilized through the presentation of demonstration examples without modifications to the model parameters. Although interesting, privacy concerns have become a major obstacle in its widespread usage. Multiple studies have examined the privacy risks linked to ICL and prompting in general, and have devised techniques to alleviate these risks. Thus, there is a necessity to organize these mitigation techniques for the benefit of the community. This survey provides a systematic overview of the privacy protection methods employed during ICL and prompting in general. We review, analyze, and compare different methods under this paradigm. Furthermore, we provide a summary of the resources accessible for the development of these frameworks. Finally, we discuss the limitations of these frameworks and offer a detailed examination of the promising areas that necessitate further exploration.

Figures (8)

• Figure 1: The layout of privacy mechanisms employed for privacy-preserving prompting. Each privacy mechanism protects at least one privacy target. We elaborate on this by creating links between the mechanisms and the privacy targets. Demo Ex(s) denotes demonstration example(s), Obfus denotes obfuscation, and CDP-FL denotes client data protection via federated learning.
• Figure 2: An illustration of ICL. In ICL, the prompt consists of a task description (light green), demonstration examples (light blue), and a query (light orange).
• Figure 3: An illustration of privatizing sensitive local data with LDP before using it to prompt a cloud LLM privately.
• Figure 4: Privacy-preserving demonstration example generation with LDP-TabICL and GDP-TabICL approaches. For LDP-TabICL (top-left), users perturb their data with the randomized response LDP mechanism before being collected. The collected data is then reconstructed to recover the original data distribution. $k$ samples are selected and serialized into text. Meanwhile, for GDP-TabICL (bottom-left), user data is collected in clear. Then the collected data is partitioned into $k$ disjoint subsets. GDP averages for each attribute in each subset are generated. The generated noisy attributes are then serialized into text. During ICL an LLM is prompted with $k$ demonstration examples selected from the serialized text and a query from a user. LLM's response is generated and sent to the user.
• Figure 5: An illustration of privacy-preserving demonstration examples generated with a local LLM, then leveraging the examples with a query to perform ICL using a cloud LLM.

Theorems & Definitions (8)

• Definition 1: Global DP
• Definition 2: Local DP
• Definition 3: Metric LDP
• Proposition 1: Post-processing property
• Proposition 2: Composition property
• Proposition 3: Amplification Effect of Sampling DBLP:conf/ccs/LiQS12
• Proposition 4: Laplace Mechanism
• Proposition 5: Exponential Mechanism