Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attacks

TL;DR

This paper tackles the problem of robustly evaluating LLM safety against jailbreaks without relying on gradients or handcrafted priors. It introduces RAILS, a gradient-free, tokenizer-agnostic framework that uses an auto-regressive loss and a history-based candidate selection to bridge the gap between proxy objectives and true harmful outcomes. Through cross-tokenizer ensemble attacks, RAILS achieves high attack success rates on open-source models and strong transfer to closed-source systems, demonstrating that shared vulnerabilities exist across disjoint vocabularies. The results highlight the need for tokenizer-agnostic defenses and provide a practical tool for automated red teaming, with significant implications for improving safety safeguards in real-world LLM deployments.

Abstract

As Large Language Models (LLMs) are increasingly deployed in safety-critical domains, rigorously evaluating their robustness against adversarial jailbreaks is essential. However, current safety evaluations often overestimate robustness because existing automated attacks are limited by restrictive assumptions. They typically rely on handcrafted priors or require white-box access for gradient propagation. We challenge these constraints by demonstrating that token-level iterative optimization can succeed without gradients or priors. We introduce RAILS (RAndom Iterative Local Search), a framework that operates solely on model logits. RAILS matches the effectiveness of gradient-based methods through two key innovations: a novel auto-regressive loss that enforces exact prefix matching, and a history-based selection strategy that bridges the gap between the proxy optimization objective and the true attack success rate. Crucially, by eliminating gradient dependency, RAILS enables cross-tokenizer ensemble attacks. This allows for the discovery of shared adversarial patterns that generalize across disjoint vocabularies, significantly enhancing transferability to closed-source systems. Empirically, RAILS achieves near 100% success rates on multiple open-source models and high black-box attack transferability to closed-source systems like GPT and Gemini.

Figures (9)

• Figure 1: RAILS: A tokenizer-agnostic and gradient-free framework. Unlike gradient-based methods which are bound to specific model weights and tokenizers, RAILS utilizes logit-based feedback from multiple open-source LLMs to generate robust adversarial prompts that effectively transfer to black-box models.
• Figure 2: The two-stage RAILS optimization process. (1) In the iterative search loop, RAILS perturbs the current best string, evaluates candidates using the source model(s), and updates the history buffer $\mathcal{H}$. (2) After optimization, top candidates are selected from history for a final few-shot attack.
• Figure 3: Impact of Source Ensemble Diversity on Transferability of RAILS-T Attack. We evaluate transfer ASR on closed-source targets (GPT-4, Gemini, GPT-3.5) while varying the source models used for optimization. The trend shows that attacking a diverse ensemble (e.g., Llama2+Vicuna+Qwen) yields significantly higher transferability than attacking single models.
• Figure 5: Perplexity vs ASR. RAILS maintains high attack success even under strict filters compared to GCG and BEAST.
• Figure : (a) Impact of Suffix Length