Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information

TL;DR

The paper tackles the vulnerability of large language models to adversarial prompts by proposing token-level detection that leverages per-token perplexity and neighboring-context information. It introduces two complementary methods: an optimization-based detector with fused-lasso regularization and a probabilistic graphical model detector that yields token-wise posteriors and sentence-level scores, both solvable in linear time via dynamic programming. Empirical results show perfect sequence-level detection and high token-level localization across models as small as GPT-2 124M, with larger models offering marginal gains in some metrics. The approach is practical, hardware-efficient, and provides intuitive heatmap visualizations, contributing to robustd LLM deployment and mitigation of adversarial prompt risks.

Abstract

In recent years, Large Language Models (LLM) have emerged as pivotal tools in various applications. However, these models are susceptible to adversarial prompt attacks, where attackers can carefully curate input strings that mislead LLMs into generating incorrect or undesired outputs. Previous work has revealed that with relatively simple yet effective attacks based on discrete optimization, it is possible to generate adversarial prompts that bypass moderation and alignment of the models. This vulnerability to adversarial prompts underscores a significant concern regarding the robustness and reliability of LLMs. Our work aims to address this concern by introducing a novel approach to detecting adversarial prompts at a token level, leveraging the LLM's capability to predict the next token's probability. We measure the degree of the model's perplexity, where tokens predicted with high probability are considered normal, and those exhibiting high perplexity are flagged as adversarial. Additionaly, our method also integrates context understanding by incorporating neighboring token information to encourage the detection of contiguous adversarial prompt sequences. To this end, we design two algorithms for adversarial prompt detection: one based on optimization techniques and another on Probabilistic Graphical Models (PGM). Both methods are equipped with efficient solving methods, ensuring efficient adversarial prompt detection. Our token-level detection result can be visualized as heatmap overlays on the text sequence, allowing for a clearer and more intuitive representation of which part of the text may contain adversarial prompts.

Figures (6)

• Figure 1: An illustrative example of token-level adversarial prompt detection. In the intermediate block, the color indicates the perplexity of each token. In the output block, the color indicates the likelihood of it being part of an adversarial prompt.
• Figure 2: Probabilistic Graphical Representation of Adversarial Prompt Detection
• Figure 3: The effect of $\lambda$ on optimization based detection.
• Figure 4: The effect of $\mu$ on optimization based detection.
• Figure 5: The effect of $\lambda$ on PGM based detection.