Language Model Inversion

TL;DR

This work formalizes prompt inversion as recovering a hidden input prefix from a language model's next-token distribution. It introduces an encoder–decoder inverter that unrolls the probability vector into a sequence of embeddings to condition a pretrained Transformer, enabling reconstruction of prompts from LM outputs with strong BLEU and F1 scores and notable exact-match recovery. The study systematically analyzes access patterns (full distributions, top-K, logit bias, text-only) and demonstrates robustness across model families, while also evaluating defenses via sampling strategies. The findings reveal meaningful privacy risks for services that expose LM outputs and propose practical defenses and avenues for future refinements in inversion techniques and mitigation strategies.

Abstract

Language models produce a distribution over the next token; can we use this information to recover the prompt tokens? We consider the problem of language model inversion and show that next-token probabilities contain a surprising amount of information about the preceding text. Often we can recover the text in cases where it is hidden from the user, motivating a method for recovering unknown prompts given only the model's current distribution output. We consider a variety of model access scenarios, and show how even without predictions for every token in the vocabulary we can recover the probability vector through search. On Llama-2 7b, our inversion method reconstructs prompts with a BLEU of $59$ and token-level F1 of $78$ and recovers $27\%$ of prompts exactly. Code for reproducing all experiments is available at http://github.com/jxmorris12/vec2text.

Figures (7)

• Figure 1: Under the assumption that a language model is offered as a service with a hidden prefix prompt that produces next-word probabilities, the system is trained from samples to invert the language model, i.e. to recover the prompt given language model probabilities for the next token.
• Figure 2: Long-term information in $\log \mathbf{v}$.
• Figure 3: Language model providers may sample differently in an effort to protect prompts from inversion. We explore inversion performance under various sampling strategies employed as defenses against inversion attacks: annealing temperature, setting top-K value, and nucleus (top-p) sampling. We consider applying temperature to the softmax both in log space (orange) and probability space (blue).
• Figure 4: (Left) Model performance under our API-based logit recovery technique vs the Monte Carlo baseline. The dotted blue line is given by reconstructing the prompt from the true probability vector. (Right) Model performance across levels of probability vector redaction. We test eliminating all except the top-K probabilities, all except the bottom-K, and all except random K, while varying K from 1 to 32,000 (full input dimensionality).
• Figure 5: Performance of jailbreak prompts by dataset.