Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Attack on Large Language Models using Exponentiated Gradient Descent

Sajib Biswas, Mao Nishino, Samuel Jacob Chacko, Xiuwen Liu

TL;DR

The paper addresses the vulnerability of open-source LLMs to jailbreaking by proposing an intrinsic adversarial attack that operates on a relaxed one-hot encoding of the vocabulary. It introduces exponentiated gradient descent on the probability simplex, combined with a KL-based Bregman projection and entropic regularization to maintain feasibility and control relaxation error, discarding the need for external projection. The approach is validated on five open-source LLMs across four adversarial datasets, outperforming three baselines in both effectiveness and efficiency, with convergence observed within a few hundred iterations. The work provides a concrete, scalable method for jailbreaking analysis and highlights considerations for model safety, transferability, and potential defenses in practical settings.

Abstract

As Large Language Models (LLMs) are widely used, understanding them systematically is key to improving their safety and realizing their full potential. Although many models are aligned using techniques such as reinforcement learning from human feedback (RLHF), they are still vulnerable to jailbreaking attacks. Some of the existing adversarial attack methods search for discrete tokens that may jailbreak a target model while others try to optimize the continuous space represented by the tokens of the model's vocabulary. While techniques based on the discrete space may prove to be inefficient, optimization of continuous token embeddings requires projections to produce discrete tokens, which might render them ineffective. To fully utilize the constraints and the structures of the space, we develop an intrinsic optimization technique using exponentiated gradient descent with the Bregman projection method to ensure that the optimized one-hot encoding always stays within the probability simplex. We prove the convergence of the technique and implement an efficient algorithm that is effective in jailbreaking several widely used LLMs. We demonstrate the efficacy of the proposed technique using five open-source LLMs on four openly available datasets. The results show that the technique achieves a higher success rate with great efficiency compared to three other state-of-the-art jailbreaking techniques. The source code for our implementation is available at: https://github.com/sbamit/Exponentiated-Gradient-Descent-LLM-Attack

Adversarial Attack on Large Language Models using Exponentiated Gradient Descent

TL;DR

The paper addresses the vulnerability of open-source LLMs to jailbreaking by proposing an intrinsic adversarial attack that operates on a relaxed one-hot encoding of the vocabulary. It introduces exponentiated gradient descent on the probability simplex, combined with a KL-based Bregman projection and entropic regularization to maintain feasibility and control relaxation error, discarding the need for external projection. The approach is validated on five open-source LLMs across four adversarial datasets, outperforming three baselines in both effectiveness and efficiency, with convergence observed within a few hundred iterations. The work provides a concrete, scalable method for jailbreaking analysis and highlights considerations for model safety, transferability, and potential defenses in practical settings.

Abstract

As Large Language Models (LLMs) are widely used, understanding them systematically is key to improving their safety and realizing their full potential. Although many models are aligned using techniques such as reinforcement learning from human feedback (RLHF), they are still vulnerable to jailbreaking attacks. Some of the existing adversarial attack methods search for discrete tokens that may jailbreak a target model while others try to optimize the continuous space represented by the tokens of the model's vocabulary. While techniques based on the discrete space may prove to be inefficient, optimization of continuous token embeddings requires projections to produce discrete tokens, which might render them ineffective. To fully utilize the constraints and the structures of the space, we develop an intrinsic optimization technique using exponentiated gradient descent with the Bregman projection method to ensure that the optimized one-hot encoding always stays within the probability simplex. We prove the convergence of the technique and implement an efficient algorithm that is effective in jailbreaking several widely used LLMs. We demonstrate the efficacy of the proposed technique using five open-source LLMs on four openly available datasets. The results show that the technique achieves a higher success rate with great efficiency compared to three other state-of-the-art jailbreaking techniques. The source code for our implementation is available at: https://github.com/sbamit/Exponentiated-Gradient-Descent-LLM-Attack
Paper Structure (24 sections, 1 theorem, 13 equations, 3 figures, 3 tables, 1 algorithm)

This paper contains 24 sections, 1 theorem, 13 equations, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Optimization Problem
  5. Exponentiated Gradient Descent
  6. Bregman Projection
  7. The Main Iteration
  8. Experiments
  9. Implementation details
  10. Models
  11. Datasets
  12. Baselines
  13. Hyper-parameters
  14. Experimental Setup
  15. Evaluation and Analysis
  16. ...and 9 more sections

Key Result

Theorem 1

For a differentiable function $F:\mathbb{R}^{L\times |\mathbb{T}|}\to \mathbb{R}$ with Lipschitz continuous gradient, eqn: main_iteration converges to a critical point (a point of zero gradient) of $F$ for small enough learning rates $\eta_t>0$.

Figures (3)

  • Figure 1: Median Cross-entropy Loss, aggregated over $50$ harmful behaviors, vs number of Epochs. With the help of Adam optimizer and regularization, EGD is able to optimize the cross-entropy loss ($\approx 0$) within the first $200$ epochs. The behavior is very consistent across all the models and datasets.
  • Figure 2: Comparison of average run-time (log-scale) of the baseline methods with our method, aggregated over all models and datasets.
  • Figure 3: Effect of the regularization terms on the sparsity of the one-hot encoding. Each curve in the two figures shows the maximal probability for each token position averaged over 50 harmful behaviors. We observe that the mean maximal probability increases significantly after introducing the regularization terms, indicating the better sparsity of the distribution.

Theorems & Definitions (3)

  • Theorem 1: Convergence
  • proof
  • Remark 1: Limitation of the convergence theorem