Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defenses

TL;DR

This work tackles prompt-injection vulnerabilities in large language models (LLMs) that are hardened with fine-tuning-based defenses. It introduces Checkpoint-GCG, a white-box attack that exploits intermediate fine-tuning checkpoints to progressively improve adversarial suffixes, including a universal suffix capable of attacking unseen prompts and transferring to similar models. The method demonstrates strong effectiveness, achieving up to $96\%$ ASR against SecAlign and $89.9\%$ ASR for universal suffixes on unseen prompts, along with notable transferability to Meta-SecAlign-8B (e.g., $63.9\%$ ASR in black-box and $78.3\%$ in white-box). These results reveal robustness gaps in current defenses and provide an auditing tool to stress-test defenses before deployment, highlighting practical risks and motivating stronger, more transferable defenses in real-world systems. The attack formalism leverages the objective $\max_s \log P_\theta(y^* \mid p || s)$ and leverages gradients to refine $s$, while Checkpoint-GCG propagates safety by exploiting the incremental nature of fine-tuning, suggesting a need for defense strategies that accommodate evolving model parameters across checkpoints.

Abstract

Large language models (LLMs) are increasingly deployed in real-world applications ranging from chatbots to agentic systems, where they are expected to process untrusted data and follow trusted instructions. Failure to distinguish between the two poses significant security risks, exploited by prompt injection attacks, which inject malicious instructions into the data to control model outputs. Model-level defenses have been proposed to mitigate prompt injection attacks. These defenses fine-tune LLMs to ignore injected instructions in untrusted data. We introduce Checkpoint-GCG, a white-box attack against fine-tuning-based defenses. Checkpoint-GCG enhances the Greedy Coordinate Gradient (GCG) attack by leveraging intermediate model checkpoints produced during fine-tuning to initialize GCG, with each checkpoint acting as a stepping stone for the next one to continuously improve attacks. First, we instantiate Checkpoint-GCG to evaluate the robustness of the state-of-the-art defenses in an auditing setup, assuming both (a) full knowledge of the model input and (b) access to intermediate model checkpoints. We show Checkpoint-GCG to achieve up to $96\%$ attack success rate (ASR) against the strongest defense. Second, we relax the first assumption by searching for a universal suffix that would work on unseen inputs, and obtain up to $89.9\%$ ASR against the strongest defense. Finally, we relax both assumptions by searching for a universal suffix that would transfer to similar black-box models and defenses, achieving an ASR of $63.9\%$ against a newly released defended model from Meta.

Figures (10)

• Figure 1: An example prompt containing both injected instruction (red) and adversarial suffix (blue).
• Figure 2: The probability of a successful attack by GCG and Checkpoint-GCG when attacking one sample on Llama-3-8B-Instruct llama3modelcard defended with SecAlign chen2024aligning.
• Figure 3: Attack Success Rate (%) against increasingly stronger defenses (Undefended, StruQ, SecAlign) across three models (Llama-3-8B-Instruct, Mistral-7B-Instruct, and Qwen2-1.5B-Instruct). Results are aggregated for $50$ randomly selected samples from AlpacaFarm.
• Figure 4: Universality of the Checkpoint-GCG suffixes on (a) in-distribution and (b) out-of-distribution test samples. Results for SecAlign; for StruQ see Appendix \ref{['app:universal_results']}.
• Figure 5: Universality of suffixes discovered by Checkpoint-GCG on (a) in-distribution and (b) out-of-distribution test samples. Results shown are for suffixes discovered against StruQ.