Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning

TL;DR

This work addresses prompt injection vulnerabilities in LLM-enabled applications by introducing InstruCoT, which combines diverse synthetic training data with instruction-level chain-of-thought guidance to detect and reject malicious instructions. The methodology comprises three phases: diverse synthesis of injection data across Behavior Deviation, Privacy Leakage, and Harmful Output; instruction-aware CoT generation that follows a three-stage reasoning process; and supervised fine-tuning to embed this reasoning into the model. Empirical results show that InstruCoT substantially improves Defense Rates across Behavior Deviation (≈92.5%), Privacy Leakage (≈98.0%), and Harmful Output (≈90.9%) over multiple baselines while maintaining or enhancing utility (Win Rate ≈83%). The approach provides a scalable defense against multi-vector PI attacks and ambiguous context boundaries, with public dataset and code to facilitate adoption and further research.

Abstract

Large language model (LLM)-integrated applications have become increasingly prevalent, yet face critical security vulnerabilities from prompt injection (PI) attacks. Defending against PI attacks faces two major issues: malicious instructions can be injected through diverse vectors, and injected instructions often lack clear semantic boundaries from the surrounding context, making them difficult to identify. To address these issues, we propose InstruCoT, a model enhancement method for PI defense that synthesizes diverse training data and employs instruction-level chain-of-thought fine-tuning, enabling LLMs to effectively identify and reject malicious instructions regardless of their source or position in the context. We evaluate InstruCoT across three critical dimensions: Behavior Deviation, Privacy Leakage, and Harmful Output. Experimental results across four LLMs demonstrate that InstruCoT significantly outperforms baselines in all dimensions while maintaining utility performance without degradation

Figures (6)

• Figure 1: The issues for the defending prompt injection attacks.
• Figure 2: The overview of InstruCoT.
• Figure 3: Analysis of prompt injection attack scenarios across different LLM applications. From outer to inner layers: LLM-based application frameworks, application components, and context regions fed to the LLM.
• Figure 4: Utility performance comparison across different methods on four LLMs.
• Figure 5: The example of InstruCoT and role-level alignment method output.