Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks

TL;DR

Open-weight LLMs remain vulnerable to prefilling attacks that bias initial responses. The paper presents the largest empirical evaluation to date across 50 models and 23 strategies, showing high attack effectiveness across model families and that size alone does not confer robust protection. It distinguishes model-agnostic from strategy-specific prefills, finding that tailored prefills can dramatically increase attack success and output detail, especially for reasoning models. These findings underscore the urgent need for stronger internal safeguards and defense strategies to mitigate prefilling in open-weight LLM deployments.

Abstract

As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on external defenses, open-weight models must primarily depend on internal safeguards to mitigate harmful behavior. Prior red-teaming research has largely focused on input-based jailbreaking and parameter-level manipulations. However, open-weight models also natively support prefilling, which allows an attacker to predefine initial response tokens before generation begins. Despite its potential, this attack vector has received little systematic attention. We present the largest empirical study to date of prefill attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, revealing a critical and previously underexplored vulnerability with significant implications for deployment. While certain large reasoning models exhibit some robustness against generic prefilling, they remain vulnerable to tailored, model-specific strategies. Our findings underscore the urgent need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.

Figures (7)

• Figure 1: Overview of the prefill attack concept.Left: Most LLMs refuse harmful requests when directly prompted under standard conditions. Right: When an attacker uses a prefill by predefining the beginning of the model’s response (red box), the model continues with a compliant response, providing the attacker with the requested information instead of refusing.
• Figure 2: Attack success rates (ASR) for recent models under standard and prefill attacks. All evaluated models are vulnerable to prefill attacks. Even models largely robust to harmful requests under standard prompting become highly susceptible, with near-perfect ASR for most models when all prefill strategies are available.
• Figure 3: Increasing model size does not improve robustness to prefill attacks. Comparison of $\text{ASR}_{\mathrm{any}}$ across models of different sizes within the same family shows that, in general, all models exhibit similar vulnerability, independent of their parameter count. The only exception is Qwen3-2507 Thinking, for which robustness decreases with model size.
• Figure 4: Attack success varies substantially across prefill strategies and models. Shown are the three most and least effective prefill attacks: some strategies achieve high success across multiple models, while others are only effective against specific models, highlighting differences in robustness among different LLMs.
• Figure 6: Attack success rates (ASR) of individual strategies on ClearHarm across the largest models from each model family. Prefill strategies are ordered by their mean ASR across all models. Some strategies consistently achieved high success, while others performed poorly across most model families. Notably, System Simulation, Fake Citation, and Continuation Full yielded the highest ASRs. For GPT-OSS, we used the prefilling strategy that skips the analysis channel.