Sockpuppetting: Jailbreaking LLMs Without Optimization Through Output Prefix Injection

TL;DR

The paper investigates sockpuppetting, a low-cost output-prefix injection attack that jailbreaks open-weight LLMs by inserting an acceptance sequence into the assistant output, bypassing gradient-based optimization. It introduces sockpuppetting as a simple baseline and demonstrates up to $80\%$ ASR improvements over traditional GCG on several models, with a further $64\%$ gain when combined in a gradient-optimized hybrid. The work shows that placing attack content in the assistant block often yields stronger universal attacks and highlights model-specific resistance (e.g., Gemma) as a challenge for defense design. This study underlines the need for defenses against output-prefix injection in open-weight models and informs both practitioners and researchers about practical, low-cost jailbreak avenues and their implications for model safety.

Abstract

As open-weight large language models (LLMs) increase in capabilities, safeguarding them against malicious prompts and understanding possible attack vectors becomes ever more important. While automated jailbreaking methods like GCG [Zou et al., 2023] remain effective, they often require substantial computational resources and specific expertise. We introduce "sockpuppetting'', a simple method for jailbreaking open-weight LLMs by inserting an acceptance sequence (e.g., "Sure, here is how to...'') at the start of a model's output and allowing it to complete the response. Requiring only a single line of code and no optimization, sockpuppetting achieves up to 80% higher attack success rate (ASR) than GCG on Qwen3-8B in per-prompt comparisons. We also explore a hybrid approach that optimizes the adversarial suffix within the assistant message block rather than the user prompt, increasing ASR by 64% over GCG on Llama-3.1-8B in a prompt-agnostic setting. The results establish sockpuppetting as an effective low-cost attack accessible to unsophisticated adversaries, highlighting the need for defences against output-prefix injection in open-weight models.

Figures (4)

• Figure 1: An overview of the attacks explored in this paper. (a) Under normal circumstances, well-aligned LLMs refuse harmful requests. (b) Gradient-guided attacks create an adversarial suffix in the prompt that gets the model to agree, by back-propagating from a predetermined acceptance sequence to the suffix. (c) Instead of solving an entire discrete optimization problem, sockpuppetting simply inserts the acceptance sequence into memory, as if the model already generated it. (d) We also combine gradient-based methods with sockpuppetting, showing that optimizing the adversarial suffix inside the "assistant" message block can increase the attack effectiveness of GCG.
• Figure 2: Sockpuppetting vastly outperforms GCG in individual attack success rates. GCG is run separately on each prompt. The sockpuppetting attacks use variations of the acceptance sequence of each prompt by inserting it into the "assistant" message block.
• Figure 3: Universal attacks are broadly more effective when conducted in the "assistant" message block. All attacks are trained on the first 25 prompts and validated on the following 100, with "Prompt only" recounted for the same validation set. The GCG and SockpuppetGCG optimize the attack suffix at once, while their rolling variants optimize successive lengths using warm start. The sockpuppet attacks optimize the attack suffixes for the "assistant" message block, while the pure GCG attacks operate inside the "user" block.
• Figure 4: Rolling universal attack success rates with different target sequences.