Soft Instruction De-escalation Defense

TL;DR

Prompt injections threaten tool-augmented LLM agents that process external data. The paper introduces Soft Instruction Control (SIC), a modular iterative preprocessor that rewrite/masks/removes instruction-like content in untrusted inputs and validates via multi-granularity checks, operating outside the agent's context. SIC is designed to be simple, cheap, and robust against a large class of prompts while remaining compatible with existing agents. Empirical results on AgentDojo across multiple models show SIC achieving zero attack success rate with minimal utility loss, though the study notes residual vulnerabilities under strong adaptive adversaries and discusses practical limitations and future directions.

Abstract

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an external environment; this makes them susceptible to prompt injections when dealing with untrusted data. To overcome this limitation, we propose SIC (Soft Instruction Control)-a simple yet effective iterative prompt sanitization loop designed for tool-augmented LLM agents. Our method repeatedly inspects incoming data for instructions that could compromise agent behavior. If such content is found, the malicious content is rewritten, masked, or removed, and the result is re-evaluated. The process continues until the input is clean or a maximum iteration limit is reached; if imperative instruction-like content remains, the agent halts to ensure security. By allowing multiple passes, our approach acknowledges that individual rewrites may fail but enables the system to catch and correct missed injections in later steps. Although immediately useful, worst-case analysis shows that SIC is not infallible; strong adversary can still get a 15% ASR by embedding non-imperative workflows. This nonetheless raises the bar.

Figures (14)

• Figure 1: Soft Instruction Control (SIC) sanitization loop for tool-augmented LLM agents. Untrusted input from the web is first augmented with dummy instructions. The system then masks, rewrites, or removes instructions within this combined data. It checks if any instructions remain; if so, it is flagged as an attack; process -- repeats. The untrusted data with sanitized instructions is then combined with the original user instruction and passed to the model as the query.
• Figure 1: Pseudo-code for SIC. The algorithm rewrites tool outputs with an LLM, then detects prompt injection in both the full and chunked outputs. If injection is detected or the control string survives, the output is halted; otherwise, a cleaned version is returned.
• Figure 2: Overall evaluation of SIC across four models (Qwen3-32B, GPT-4o, Kimi-k2, GPT-4.1-mini) and three suites (banking, slack, travel). (a) Baseline (b) SIC (masking, one pass, no chunking). SIC reduces ASR to $0$ with little utility loss. (c) RUP. ASR improves compared to baseline but is worse than for SIC. (d) SWD. Has the highest utility under attack, while having the highest ASR.
• Figure 3: Utility and ASR for GPT-4.1-mini and Qwen3-32b across three sanitization strategies on clean inputs (with/without defense) and adversarial inputs (with defense). Mask best preserves utility while eliminating ASR.
• Figure 4: Ablations. (a) 1--3 rewrite passes: small robustness gains at the cost of clean utility (not needed for current attacks). (b) Chunked classification: higher false positives reduce utility but maintain 0% ASR in our setting.