Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stealing Part of a Production Language Model

Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, Florian Tramèr

TL;DR

This work demonstrates a practical, top-down model-stealing approach that extracts the embedding projection layer of production transformers from black-box LLM APIs. By exploiting logit-vector and logit-bias outputs, the authors recover the final layer weights up to symmetries, revealing hidden dimensions (e.g., 1024 and 2048) and enabling high-fidelity reconstruction on multiple models, including OpenAI Ada, Babbage, and GPT-3.5-turbo variants. The paper also extends attacks to constrained APIs (top-K logits and logit-bias) and logprob-free settings, evaluates on production-like models, and proposes defenses (removing logit bias, architectural changes, noise, rate limits) and future research directions. The results underscore the practicality of partial parameter leakage from deployed LLMs and motivate design choices that limit such information exposure while balancing functionality.

Abstract

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

Stealing Part of a Production Language Model

TL;DR

This work demonstrates a practical, top-down model-stealing approach that extracts the embedding projection layer of production transformers from black-box LLM APIs. By exploiting logit-vector and logit-bias outputs, the authors recover the final layer weights up to symmetries, revealing hidden dimensions (e.g., 1024 and 2048) and enabling high-fidelity reconstruction on multiple models, including OpenAI Ada, Babbage, and GPT-3.5-turbo variants. The paper also extends attacks to constrained APIs (top-K logits and logit-bias) and logprob-free settings, evaluates on production-like models, and proposes defenses (removing logit bias, architectural changes, noise, rate limits) and future research directions. The results underscore the practicality of partial parameter leakage from deployed LLMs and motivate design choices that limit such information exposure while balancing functionality.

Abstract

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.
Paper Structure (66 sections, 11 theorems, 42 equations, 7 figures, 4 tables, 3 algorithms)

This paper contains 66 sections, 11 theorems, 42 equations, 7 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Extraction Attack for Logit-Vector APIs
  5. Warm-up: Recovering Hidden Dimensionality
  6. Cheaper Dimension Extraction
  7. Full Layer Extraction (Up to Symmetries)
  8. Experiments.
  9. Extraction Attack for Logit-Bias APIs
  10. Description of the API
  11. Evaluation Methodology
  12. Extraction Attack for Top-5 Logit Bias APIs
  13. An improved cost-optimal attack.
  14. Extraction Attack for top-1 Binary Logit Bias APIs
  15. Logprob-free attacks
  16. ...and 51 more sections

Key Result

Lemma 4.1

Let $\mathbf{Q}\left({p_1, \ldots p_{n}}\right) \in \mathbb{R}^{l \times n}$ denote the matrix with columns $\mathcal{O}\left({p_1}\right), \ldots, \mathcal{O}\left({p_{n}}\right)$ of query responses from the logit-vector API. Then Further, if the matrix with columns $g_\theta\left({p_i}\right)$ ($i=1, ..., n$) has rank $h$ and $\mathbf{W}$ has rank $h$, then

Figures (7)

  • Figure 1: SVD can recover the hidden dimensionality of a model when the final output layer dimension is greater than the hidden dimension. Here we extract the hidden dimension (2048) of the Pythia 1.4B model. We can precisely identify the size by obtaining slightly over 2048 full logit vectors.
  • Figure 2: Our extraction attack recovers the hidden dimension by identifying a sharp drop in singular values, visualized as a spike in the difference between consecutive singular values. On Pythia-1.4B, a 2048 dimensional model, the spike occurs at 2047 values.
  • Figure 4: Detecting whether models use LayerNorm or RMSNorm by singular value magnitudes.
  • Figure 5: Stress-testing the LayerNorm extraction attack on models behind an API (a-b), and models using both RMSNorm and biases (c).
  • Figure 7: On the left, we plot the singular values that are extracted using our attack on GPT-2 small---the estimated hidden dimension is near 768. On the right, we post-hoc extend the dimensionality of the weight matrix to 1024, as described in Section \ref{['sec:defense']}. This misleads the adversary into thinking the model is wider than it actually is.
  • ...and 2 more figures

Theorems & Definitions (21)

  • Lemma 4.1
  • proof
  • Lemma 4.2
  • proof
  • proof : Proof of \ref{['lemma:recover-E-up-to-rotation']}(ii)
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof : Proof of Lemma \ref{['lemma:linear-algebra-orthogonal']}
  • Lemma 6.1
  • ...and 11 more