Vocabulary Attack to Hijack Large Language Model Applications

TL;DR

The paper addresses jailbreak vulnerabilities in LLM-based applications by introducing a vocabulary-based prompt attack that inserts adversarial words drawn from an attacker LLM's vocabulary to achieve goal hijacking. The method optimizes a loss combining output similarity to a desired target and token-length constraints, inserting up to three words across 10 epochs at arbitrary prompt positions. Experiments on open-source models (Llama2-7B-CHAT-HF and FLAN-T5-XXL) with attacker models (Llama2-CHAT-HF and T5-BASE) show that single-word insertions can often succeed and that the attack transfers across attacker/target model configurations, sometimes outperforming traditional separator-based approaches in stealth. The findings highlight practical security risks for LLM applications and motivate improved prompt defenses and automated robustness testing across diverse models and attack goals.

Abstract

The fast advancements in Large Language Models (LLMs) are driving an increasing number of applications. Together with the growing number of users, we also see an increasing number of attackers who try to outsmart these systems. They want the model to reveal confidential information, specific false information, or offensive behavior. To this end, they manipulate their instructions for the LLM by inserting separators or rephrasing them systematically until they reach their goal. Our approach is different. It inserts words from the model vocabulary. We find these words using an optimization procedure and embeddings from another LLM (attacker LLM). We prove our approach by goal hijacking two popular open-source LLMs from the Llama2 and the Flan-T5 families, respectively. We present two main findings. First, our approach creates inconspicuous instructions and therefore it is hard to detect. For many attack cases, we find that even a single word insertion is sufficient. Second, we demonstrate that we can conduct our attack using a different model than the target model to conduct our attack with.