GPT, But Backwards: Exactly Inverting Language Model Outputs

TL;DR

This work reframes recovering the original input to a language model from its output as a discrete optimization problem and introduces SODA, a white-box search employing a continuous relaxation and a modified Adam optimizer. SODA achieves markedly higher exact reconstruction rates than prior methods, including robust performance on random inputs up to length 10, and scales across models from 33M to 3B parameters. The authors formalize the objective, design initialization and relaxation mechanisms, and demonstrate practical uses in privacy leakage, backdoor detection, and slander verification. The findings show that input length and vocabulary size drive inversion feasibility, that access to logit information substantially boosts reconstruction, and that search-based inversion can outperform learned inversion approaches in challenging auditing scenarios.

Abstract

The task of reconstructing unknown textual inputs to language models is a fundamental auditing primitive that allows us to assess the model's vulnerability to a range of security issues, including stealing hidden system prompts, detecting backdoors, and leaking private data. Existing inversion works assume access to differing levels of information (e.g. requiring input-output examples, the model parameters, intermediate activations or output logits) but oftentimes fail to fully reconstruct the desired input. In this paper, we present the Sparse One-hot Discrete Adam (SODA) algorithm, a search-based inversion method that can accurately reconstruct the input text, given white-box access to the language model and its output. Our experiments demonstrate for the first time that exact language model inversion is possible on both natural language and random inputs. Indeed, SODA achieves respectively 98% and 79% reconstruction rates on inputs with lengths up to 10 tokens. Furthermore, we show that input length and vocabulary size have a far greater impact on the probability of a successful reconstruction than the size of the language model itself, thus allowing us to scale to models from 33M to 3B parameters.

Figures (6)

• Figure 1: Existing techniques fail to reconstruct the original input $x$ to the LLM: some can only reconstruct a different input $x'\neq x$ such that $f(x') = y$, as with adversarial approaches shi2022human, others do not even achieve this goal ($f(x')\neq y$). The input-output pair is a real example from the TinyStories-33M model and so are the inputs recovered by each method.
• Figure 2: High-level diagram of LLM generation.
• Figure 3: Percentage of exact matches found by SODA over iterations of search, broken down by the lengths of inputs inverted.
• Figure 4: Percentage of exactly matching tokens found by SODA at specific positions in the input sequence, broken down by the lengths of inputs inverted.
• Figure 5: Token highlighting is present in LLM responses when using the llama.cpp interface for running open source models, reflecting the probabilities of those tokens being sampled during generation, potentially exposing a new attack surface.