Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt Inversion Attack against Collaborative Inference of Large Language Models

Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, Jiaheng Zhang

TL;DR

This paper analyzes privacy risks in collaborative inference for large language models by introducing the prompt inversion attack (PIA), where a malicious participant recovers input prompts from intermediate activations. It develops a white-box PIA based on constrained optimization and adaptive discretization, and extends it to grey-box scenarios with LoRA adapters. Experimental results across multiple datasets and models show near-perfect prompt recovery in the white-box setting (e.g., $88.4\%$ token accuracy on Skytrax with Llama-65B) and strong performance in grey-box settings (≈$85\%$ token accuracy), highlighting significant privacy threats. The work also evaluates defenses, finding classical approaches like activation obfuscation or quantization generally insufficient, thereby motivating new defense mechanisms and further research into privacy-preserving collaborative LLM inference.

Abstract

Large language models (LLMs) have been widely applied for their remarkable capability of content generation. However, the practical use of open-source LLMs is hindered by high resource requirements, making deployment expensive and limiting widespread development. The collaborative inference is a promising solution for this problem, in which users collaborate by each hosting a subset of layers and transmitting intermediate activation. Many companies are building collaborative inference platforms to reduce LLM serving costs, leveraging users' underutilized GPUs. Despite widespread interest in collaborative inference within academia and industry, the privacy risks associated with LLM collaborative inference have not been well studied. This is largely because of the challenge posed by inverting LLM activation due to its strong non-linearity. In this paper, to validate the severity of privacy threats in LLM collaborative inference, we introduce the concept of prompt inversion attack (PIA), where a malicious participant intends to recover the input prompt through the activation transmitted by its previous participant. Extensive experiments show that our PIA method substantially outperforms existing baselines. For example, our method achieves an 88.4\% token accuracy on the Skytrax dataset with the Llama-65B model when inverting the maximum number of transformer layers, while the best baseline method only achieves 22.8\% accuracy. The results verify the effectiveness of our PIA attack and highlights its practical threat to LLM collaborative inference systems.

Prompt Inversion Attack against Collaborative Inference of Large Language Models

TL;DR

This paper analyzes privacy risks in collaborative inference for large language models by introducing the prompt inversion attack (PIA), where a malicious participant recovers input prompts from intermediate activations. It develops a white-box PIA based on constrained optimization and adaptive discretization, and extends it to grey-box scenarios with LoRA adapters. Experimental results across multiple datasets and models show near-perfect prompt recovery in the white-box setting (e.g., token accuracy on Skytrax with Llama-65B) and strong performance in grey-box settings (≈ token accuracy), highlighting significant privacy threats. The work also evaluates defenses, finding classical approaches like activation obfuscation or quantization generally insufficient, thereby motivating new defense mechanisms and further research into privacy-preserving collaborative LLM inference.

Abstract

Large language models (LLMs) have been widely applied for their remarkable capability of content generation. However, the practical use of open-source LLMs is hindered by high resource requirements, making deployment expensive and limiting widespread development. The collaborative inference is a promising solution for this problem, in which users collaborate by each hosting a subset of layers and transmitting intermediate activation. Many companies are building collaborative inference platforms to reduce LLM serving costs, leveraging users' underutilized GPUs. Despite widespread interest in collaborative inference within academia and industry, the privacy risks associated with LLM collaborative inference have not been well studied. This is largely because of the challenge posed by inverting LLM activation due to its strong non-linearity. In this paper, to validate the severity of privacy threats in LLM collaborative inference, we introduce the concept of prompt inversion attack (PIA), where a malicious participant intends to recover the input prompt through the activation transmitted by its previous participant. Extensive experiments show that our PIA method substantially outperforms existing baselines. For example, our method achieves an 88.4\% token accuracy on the Skytrax dataset with the Llama-65B model when inverting the maximum number of transformer layers, while the best baseline method only achieves 22.8\% accuracy. The results verify the effectiveness of our PIA attack and highlights its practical threat to LLM collaborative inference systems.

Paper Structure

This paper contains 30 sections, 2 theorems, 34 equations, 8 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Large Language Model (LLM)
  4. Collaborative Inference System of LLMs
  5. Inversion Attacks in Machine Learning
  6. Threat Model and Problem Formulation
  7. Threat Model
  8. Problem Formulation
  9. Observations and Insights
  10. Methodology
  11. Outline of Prompt Inversion Attack
  12. Constrained Optimization
  13. Adaptive Discretization
  14. Grey-box Prompt Inversion Attack
  15. Experiments
  16. ...and 15 more sections

Key Result

Theorem 1

For vocabulary set $V$, embedding hidden dimension $h$, token embedding matrix $W:\mathbb{R}^{|V|\times h}$, embedding vector $v: \mathbb{R}^{h}$ , word selection weight $z:\mathbb{R}^{|V|}$, optimization target value $L$, $v=W^\top softmax(z)$. Let $s=softmax(z)$. Assume $W_{ij} \sim \mathcal{N}(0

Figures (8)

  • Figure 1: Outline of our attack setting. In collaborative inference, the LLM’s transformer blocks are distributed across collaborating participants, each holding a portion of the model. In this figure, 'ACT' represents the intermediate activation transmitted between participants. User 3 is a malicious participant who records the intermediate activation received from User 2, completing the computation as usual. After inference, User 3 then attempts to reconstruct the input prompt using the recorded activation.
  • Figure 2: The distribution of $|\frac{\partial L}{\partial z}|$ and $|\frac{\partial L}{\partial \hat{v}}|$.
  • Figure 3: Workflow of our prompt inversion attack. (1) The attacker first conducts constrained optimization to approximately recover the input embedding, and obtain the embedding-distance based candidate set $S_e$. Our designed constraint improves the result quality of embedding optimization. The attacker also computes a semantic-based candidate set $S_s$, obtained by feeding previously recovered tokens into an oracle LLM. (2)$S_e$ and $S_s$ are then joined. (3) Activation calibration. (4) The attacker enumerates the tokens in $S_e \bigcup S_s$ and finds the token that best matches the target activation.
  • Figure 4: Value distribution of ground-truth embedding $E(x)$ and optimized embedding $\hat{v}$ obtained by solely optimizing based on Equation (\ref{['eq:easier version']}) on Llama-7B model. We can observe that $\hat{v}$ is far away from $E(x)$.
  • Figure 5: Two examples of our prompt inversion attack. The first example prompt is an airline review from Skytrax dataset, while the second is from ECHR Law dataset. Tokens inverted correctly are marked in green, while the wrong ones are marked in red.
  • ...and 3 more figures

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2
  • proof
  • proof