Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers

TL;DR

The paper examines security vulnerabilities in LLM-based prompt optimization by identifying two attack surfaces: Query Manipulation and Feedback Manipulation. It introduces a novel fake reward attack that exploits optimization feedback loops and demonstrates its effectiveness on HarmBench using GPT-4.1 as both optimizer and backend, finding substantial increases in attack success rate ($\Delta$ASR) when feedback is manipulated. A lightweight highlighting defense is proposed, which reduces $\Delta$ASR from $0.23$ to $0.07$ without harming utility. The work highlights prompt optimization as a distinct attack surface and calls for stronger safeguards around feedback channels and optimization pipelines, outlining future directions for domain-specific and multimodal defenses.

Abstract

Large language model (LLM) systems now underpin everyday AI applications such as chatbots, computer-use assistants, and autonomous robots, where performance often depends on carefully designed prompts. LLM-based prompt optimizers reduce that effort by iteratively refining prompts from scored feedback, yet the security of this optimization stage remains underexamined. We present the first systematic analysis of poisoning risks in LLM-based prompt optimization. Using HarmBench, we find systems are substantially more vulnerable to manipulated feedback than to injected queries: feedback-based attacks raise attack success rate (ASR) by up to $Δ$ASR = 0.48. We introduce a simple fake-reward attack that requires no access to the reward model and significantly increases vulnerability, and we propose a lightweight highlighting defense that reduces the fake-reward $Δ$ASR from 0.23 to 0.07 without degrading utility. These results establish prompt optimization pipelines as a first-class attack surface and motivate stronger safeguards for feedback channels and optimization frameworks.

Figures (3)

• Figure 1: System Messages: Initialization and Optimization Under Attacks/Defenses. 1) Initial System Message, the baseline system message defined before optimization; 2) Direct Feedback Manipulation, an attacker leverages the harmscore as scalar feedback to misguide the LLM optimizer; 3) Fake Feedback Attack, exploits the optimizer's vulnerability by providing positive feedback on harmful responses at end of query, to fool the system as genuine feedback; 4) Defense via Highlighting, highlights potentially vulnerable parts of the query, making the optimizer aware of and resistant to fake feedback reward attacks.
• Figure 2: Vulnerabilities in LLM-based Prompt Optimizers. We identify two main sources of vulnerability in LLM-based prompt optimization. First is Query Manipulation, where an attacker introduces harmful queries that cause the resulting system prompt to become more vulnerable or behave in undesirable ways. Second is Feedback Manipulation, where an attacker gains control over the feedback source and uses it to manipulate the final system prompt for malicious purposes.
• Figure 3: Training Dynamics and Harmful System Prompt. We present the training dynamics of the harmscore feedback and fake reward attack experiments, showing how the optimized metric and ASR evolve over time. We also highlight parts of the most harmful system prompt observed in message boxes.