Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models

TL;DR

Contextual priming can steer LLMs toward unsafe outputs. The paper formalizes Response Attack (RA), a two-stage jailbreak that uses an auxiliary model to craft an initial prompt, generate a mildly harmful intermediate reply, and then issue a trigger prompt to induce explicit harmful content from the target model. RA employs Direct (DRI) and Scaffolding (SRI) Response Injection, achieving markedly higher attack success rates than nine baselines across eight state-of-the-art LLMs, while maintaining semantic fidelity and efficiency. The work also analyzes defenses, showing partial robustness of current methods and recommending defense strategies based on contextual-priming-aware fine-tuning with refusal data to mitigate this vulnerability while preserving model helpfulness.

Abstract

Contextual priming, where earlier stimuli covertly bias later judgments, offers an unexplored attack surface for large language models (LLMs). We uncover a contextual priming vulnerability in which the previous response in the dialogue can steer its subsequent behavior toward policy-violating content. While existing jailbreak attacks largely rely on single-turn or multi-turn prompt manipulations, or inject static in-context examples, these methods suffer from limited effectiveness, inefficiency, or semantic drift. We introduce Response Attack (RA), a novel framework that strategically leverages intermediate, mildly harmful responses as contextual primers within a dialogue. By reformulating harmful queries and injecting these intermediate responses before issuing a targeted trigger prompt, RA exploits a previously overlooked vulnerability in LLMs. Extensive experiments across eight state-of-the-art LLMs show that RA consistently achieves significantly higher attack success rates than nine leading jailbreak baselines. Our results demonstrate that the success of RA is directly attributable to the strategic use of intermediate responses, which induce models to generate more explicit and relevant harmful content while maintaining stealth, efficiency, and fidelity to the original query. The code and data are available at https://github.com/Dtc7w3PQ/Response-Attack.

Figures (10)

• Figure 1: Illustration of RA. (a) A harmful query is initially rejected. (b) The query elicits unsafe responses after contextual priming with injected dialogue via DRI or SRI.
• Figure 2: Overview of the proposed Response Attack (RA) framework. The RA pipeline consists of four components: Initial Prompt Crafting, Harmful Response Injection (via DRI or SRI), Trigger Prompt Construction, and final Attack Execution.
• Figure 3: Category-wise ASR performance of RA-DRI (a) and RA-SRI (b) on the six HarmBench categories: chemical biological, cybercrime intrusion, harassment bullying, harmful, illegal, and misinformation disinformation.
• Figure 4: Attack efficiency and ASR (%) comparison across three representative models. RA methods achieve higher success rates with significantly fewer interactions than baseline methods such as ActorAttack and Crescendo.
• Figure 5: Semantic similarity (cosine) between original queries and attack prompts across different methods. Higher values indicate better semantic fidelity.