An Invariant Latent Space Perspective on Language Model Inversion

TL;DR

This work reframes language model inversion as exploiting an invariant latent space shared between prompts and outputs. By formalizing the Invariant Latent Space Hypothesis (ILSH) with source invariance and cyclic invariance, the authors design Inv^2A, an end-to-end attack that reuses the forward LLM as an invariant decoder while learning a lightweight inverse encoder to map outputs into denoised latent representations. Training proceeds in two phases—contrastive alignment and supervised reinforcement—with an optional training-free post-refinement to expand local search neighborhoods; a semi-sparse encoding mechanism keeps computation tractable when multiple outputs are available. Across nine datasets covering user and system prompts, Inv^2A achieves state-of-the-art inversion performance, averaging a BLEU improvement of 4.77% over baselines and requiring far less inverse data, while exposing the limited protective value of existing defenses. The work highlights significant privacy risks in LLM deployments and calls for stronger, privacy-preserving defenses alongside responsible disclosure and evaluation practices.

Abstract

Language model inversion (LMI), i.e., recovering hidden prompts from outputs, emerges as a concrete threat to user privacy and system security. We recast LMI as reusing the LLM's own latent space and propose the Invariant Latent Space Hypothesis (ILSH): (1) diverse outputs from the same source prompt should preserve consistent semantics (source invariance), and (2) input<->output cyclic mappings should be self-consistent within a shared latent space (cyclic invariance). Accordingly, we present Inv^2A, which treats the LLM as an invariant decoder and learns only a lightweight inverse encoder that maps outputs to a denoised pseudo-representation. When multiple outputs are available, they are sparsely concatenated at the representation layer to increase information density. Training proceeds in two stages: contrastive alignment (source invariance) and supervised reinforcement (cyclic invariance). An optional training-free neighborhood search can refine local performance. Across 9 datasets covering user and system prompt scenarios, Inv^2A outperforms baselines by an average of 4.77% BLEU score while reducing dependence on large inverse corpora. Our analysis further shows that prevalent defenses provide limited protection, underscoring the need for stronger strategies. The source code and data involved in this paper can be found in https://github.com/yyy01/Invariant_Attacker.

Figures (10)

• Figure 1: Overview of $\text{Inv}^2\text{A}$. An inverse encoder maps one or more outputs into denoised pseudo-representations in the LLM's latent space, and the LLM is reused to recover the prompt. The threat model covers both user prompt and system prompt.
• Figure 2: Evaluation of cyclic invariance. Synonym Replacement, Random Swap (randomly swapping words within a sentence), and Random Noise (replacing words with random WordNet entries) represent different perturbation types. Numbers in parentheses indicate the proportion of perturbed words. The brown dashed line marks the mean under the original setting.
• Figure 3: Robustness against prompt length and synonyms.
• Figure 4: Results under varying training data scales.
• Figure 5: Importance visualization, where the prompt is "How do you feel about the current political climate in the US? What is one thing that you would like to change?"