Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TopicAttack: An Indirect Prompt Injection Attack via Topic Transition

Yulin Chen, Haoran Li, Yuexin Li, Yue Liu, Yangqiu Song, Bryan Hooi

TL;DR

This paper addresses indirect prompt injection in LLMs by introducing TopicAttack, which generates a fabricated conversational transition that gradually shifts from benign content to the injected instruction. The method combines a topic-transition prompt with an attention-maintaining reminding prompt, enabling high attack success rates (ASR) across diverse chatbots and agents, including large-open and closed-source models, even under defense mechanisms. Empirical results show ASR often exceeds 90% and correlates with higher injected-to-original attention ratios, while ablations confirm the importance of reminding prompts, multi-turn context, and robust transition design. The work provides a practical, automated approach to constructing stealthier injections and highlights the need for stronger defenses, contributing to understanding vulnerabilities in LLM-integrated systems and informing future defense strategies.

Abstract

Large language models (LLMs) have shown remarkable performance across a range of NLP tasks. However, their strong instruction-following capabilities and inability to distinguish instructions from data content make them vulnerable to indirect prompt injection attacks. In such attacks, instructions with malicious purposes are injected into external data sources, such as web documents. When LLMs retrieve this injected data through tools, such as a search engine and execute the injected instructions, they provide misled responses. Recent attack methods have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. Motivated by the limitations of existing attack methods, we propose TopicAttack, which prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction, making the injection smoother and enhancing the plausibility and success of the attack. Through comprehensive experiments, TopicAttack achieves state-of-the-art performance, with an attack success rate (ASR) over 90\% in most cases, even when various defense methods are applied. We further analyze its effectiveness by examining attention scores. We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.

TopicAttack: An Indirect Prompt Injection Attack via Topic Transition

TL;DR

This paper addresses indirect prompt injection in LLMs by introducing TopicAttack, which generates a fabricated conversational transition that gradually shifts from benign content to the injected instruction. The method combines a topic-transition prompt with an attention-maintaining reminding prompt, enabling high attack success rates (ASR) across diverse chatbots and agents, including large-open and closed-source models, even under defense mechanisms. Empirical results show ASR often exceeds 90% and correlates with higher injected-to-original attention ratios, while ablations confirm the importance of reminding prompts, multi-turn context, and robust transition design. The work provides a practical, automated approach to constructing stealthier injections and highlights the need for stronger defenses, contributing to understanding vulnerabilities in LLM-integrated systems and informing future defense strategies.

Abstract

Large language models (LLMs) have shown remarkable performance across a range of NLP tasks. However, their strong instruction-following capabilities and inability to distinguish instructions from data content make them vulnerable to indirect prompt injection attacks. In such attacks, instructions with malicious purposes are injected into external data sources, such as web documents. When LLMs retrieve this injected data through tools, such as a search engine and execute the injected instructions, they provide misled responses. Recent attack methods have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. Motivated by the limitations of existing attack methods, we propose TopicAttack, which prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction, making the injection smoother and enhancing the plausibility and success of the attack. Through comprehensive experiments, TopicAttack achieves state-of-the-art performance, with an attack success rate (ASR) over 90\% in most cases, even when various defense methods are applied. We further analyze its effectiveness by examining attention scores. We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.

Paper Structure

This paper contains 52 sections, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt Injection Attacks
  4. Prompt Injection Defenses
  5. Threat Model
  6. Attackers' Goal.
  7. Attackers' Accessibility.
  8. Attackers' Knowledge.
  9. Methodology
  10. Problem Formulation
  11. Attack via Topic Transition
  12. Topic Transition.
  13. Attention Maintenance on Injected Instruction.
  14. Experiments
  15. Experimental Settings
  16. ...and 37 more sections

Figures (3)

  • Figure 1: An example of the abrupt instruction injection (a) and our method, TopicAttack (b). We fabricate dialogue histories and inject the instruction in a way that makes the insertion smoother. "[user]" and "[assistant]" indicate whose turn it is in the conversation. "[instruction]" indicates that the following content is an instruction and it can also be used to "[data]" and "[response]" to clarify their roles. All of them are manually crafted by the attackers.
  • Figure 2: Distribution of the average log perplexity of the injected instruction within the entire input prompt.
  • Figure 3: Distribution of the attention score ratio between injected and original instructions (Inj / Ori Score Ratio) under three defense settings: No Defense, StruQ, and DPO.