SecInfer: Preventing Prompt Injection via Inference-time Scaling
Yupei Liu, Yanting Wang, Yuqi Jia, Jinyuan Jia, Neil Zhenqiang Gong
TL;DR
Prompt injection poses a critical risk to LLMs, including in AI-powered agents. SecInfer introduces inference-time scaling tailored to this threat, combining system-prompt-guided sampling to generate diverse candidate responses and target-task-guided aggregation to select the candidate best aligned with the intended task. Across six target tasks, two model families, and both existing and adaptive attacks, SecInfer significantly reduces attack success while preserving utility, outperforming prior defenses and general inference-time methods. The approach balances robustness with practical overhead, enabling deployment in real-world LLM systems and agent workloads while outlining avenues for future improvements.
Abstract
Prompt injection attacks pose a pervasive threat to the security of Large Language Models (LLMs). State-of-the-art prevention-based defenses typically rely on fine-tuning an LLM to enhance its security, but they achieve limited effectiveness against strong attacks. In this work, we propose \emph{SecInfer}, a novel defense against prompt injection attacks built on \emph{inference-time scaling}, an emerging paradigm that boosts LLM capability by allocating more compute resources for reasoning during inference. SecInfer consists of two key steps: \emph{system-prompt-guided sampling}, which generates multiple responses for a given input by exploring diverse reasoning paths through a varied set of system prompts, and \emph{target-task-guided aggregation}, which selects the response most likely to accomplish the intended task. Extensive experiments show that, by leveraging additional compute at inference, SecInfer effectively mitigates both existing and adaptive prompt injection attacks, outperforming state-of-the-art defenses as well as existing inference-time scaling approaches.