Goal-guided Generative Prompt Injection Attack on Large Language Models

TL;DR

This work targets LLM security by formulating prompt injection as maximizing the KL-divergence between posteriors under clean and adversarial inputs, and proving that under a Gaussian assumption this is equivalent to maximizing the Mahalanobis distance between embeddings. It then introduces G2PIA, a goal-guided generative prompt injection method that uses POS-based core words and an auxiliary LLM to craft adversarial prompts and insert them into the original text, all in a black-box, query-free setting. The approach leverages semantic constraints via WordNet for core words and a cosine-similarity constraint to approximate the optimal solution, yielding an effective and low-cost attack across seven LLMs and four datasets, with notable transferability. The findings highlight potential robustness gaps in current LLMs and offer a practical framework for evaluating and improving defenses against sophisticated, goal-driven prompt injections.

Abstract

Current large language models (LLMs) provide a strong foundation for large-scale user-oriented natural language tasks. A large number of users can easily inject adversarial text or instructions through the user interface, thus causing LLMs model security challenges. Although there is currently a large amount of research on prompt injection attacks, most of these black-box attacks use heuristic strategies. It is unclear how these heuristic strategies relate to the success rate of attacks and thus effectively improve model robustness. To solve this problem, we redefine the goal of the attack: to maximize the KL divergence between the conditional probabilities of the clean text and the adversarial text. Furthermore, we prove that maximizing the KL divergence is equivalent to maximizing the Mahalanobis distance between the embedded representation $x$ and $x'$ of the clean text and the adversarial text when the conditional probability is a Gaussian distribution and gives a quantitative relationship on $x$ and $x'$. Then we designed a simple and effective goal-guided generative prompt injection strategy (G2PIA) to find an injection text that satisfies specific constraints to achieve the optimal attack effect approximately. It is particularly noteworthy that our attack method is a query-free black-box attack method with low computational cost. Experimental results on seven LLM models and four datasets show the effectiveness of our attack method.

Figures (6)

• Figure 1: Assumption that the output $y$ of LLM under the condition of $x$ satisfies the discrete Gaussian distribution: Answers (output) $y$ close to question (input) $x$ are usually more relevant to $x$ and have a higher probability of being sampled.
• Figure 2: Assuming $x'^* = (x'_1,x'_2)$ is the optimal solution to problem (\ref{['eqn:opt-problem-text']}), then when $x'$ moves from $A$ through $x'^*$ to $B$ on the ellipse, $\cos(x',x)$ first increases and then decreases, while $\|x'\|_2$ first decreases and then increases.
• Figure 3: Overview of Goal-guided Generative Prompt Injection Attack: 1) We use the part-of-speech method to find the subject, predicate and object of the question in the clean text $x$ and fetch synonyms of the predicate and object plus a random number as core words; 2) Put the core words into assistant LLM to generate an adversarial text $x'$ that satisfies the constraints; 3) Insert the generated adversarial text into the clean text $x$ to form the final attack text; 4) Enter the attack text into the LLM victimization model to test the effectiveness of our attack strategy.
• Figure 4: Transfer Success Rate(TSR) heatmap. The rows and columns represent the attack model and defense model, respectively.
• Figure 5: ASR metric changes with the parameter $\epsilon$