Evil twins are not that evil: Qualitative insights into machine-generated prompts

TL;DR

The paper provides a first thorough qualitative analysis of opaque autoprompts across six decoder-only language models, revealing that autoprompts comprise a last-token with strong influence, prunable tokens, fillers, and keywords that together shape continuations. Through a gradient-based autoprompt induction and a suite of perturbations (pruning, replacement, shuffling), the study shows that many tokens can be removed or swapped without altering the outcome, yet the last token remains critical and the overall prompt exhibits partial semantic compositionality rather than being a monolithic opaque trigger. Importantly, many findings generalize to natural prompts, suggesting these effects are rooted in fundamental LM language processing and not unique to autoprompt generation. The results have practical significance for understanding model robustness, prompting design, and safety, indicating both vulnerabilities to autoprompt-based manipulation and potential avenues for resilient prompt construction.

Abstract

It has been widely observed that language models (LMs) respond in predictable ways to algorithmically generated prompts that are seemingly unintelligible. This is both a sign that we lack a full understanding of how LMs work, and a practical challenge, because opaqueness can be exploited for harmful uses of LMs, such as jailbreaking. We present the first thorough analysis of opaque machine-generated prompts, or autoprompts, pertaining to 6 LMs of different sizes and families. We find that machine-generated prompts are characterized by a last token that is often intelligible and strongly affects the generation. A small but consistent proportion of the previous tokens are prunable, probably appearing in the prompt as a by-product of the fact that the optimization process fixes the number of tokens. The remaining tokens fall into two categories: filler tokens, which can be replaced with semantically unrelated substitutes, and keywords, that tend to have at least a loose semantic relation with the generation, although they do not engage in well-formed syntactic relations with it. Additionally, human experts can reliably identify the most influential tokens in an autoprompt a posteriori, suggesting these prompts are not entirely opaque. Finally, some of the ablations we applied to autoprompts yield similar effects in natural language inputs, suggesting that autoprompts emerge naturally from the way LMs process linguistic inputs in general.

Figures (10)

• Figure 1: We analyze opaque machine-generated prompts (autoprompts) and identify four key components: (1) the last token, highly influential and difficult to modify; (2) prunable tokens, being ignored by the LM; (3) key tokens, which carry loosely related semantic information, essential for generation, and that can be replaced with semantically similar tokens; and (4) fillers, which can be substituted with a large amount of unrelated tokens but, unlike prunable tokens, cannot be deleted.
• Figure 2: Counts of tokens that were pruned (dark orange) and kept (yellow) by position for Pythia-6.9B, where 0 is the last position. Tokens at last position are extremely unlikely to be pruned.
• Figure 3: Pile-based log frequency distributions of bigrams in the original prompts, autoprompts and at the autoprompt/continuation boundary (ap/cont boundary) for Pythia-6.9B. Log(0) conventionally set to -1. Red line = median; boxes span interquartile ranges.
• Figure 4: Average proportions of replacement effect types by position on pruned prompts for Pythia-6.9B, aligned from right (whiskers show standard deviations). Null-effect replacements leave the continuation unchanged. Moderate replacements have BLEU $\geq 0.2$. Strong replacements have BLEU $<0.2$.
• Figure 5: Semantic similarity between substitutes. When a token admits fewer substitutes, they generally lie in a smaller semantic space than when the substitute count is large. The rightmost data points correspond to the most flexible tokens and tend to be semantically inconsistent. The leftmost data points are tokens admitting 0 or few substitutes, which are semantically closer.