Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Membership Inference Attacks on Tokenizers of Large Language Models

Meng Tong, Yuntao Du, Kejiang Chen, Weiming Zhang, Ninghui Li

TL;DR

This work identifies tokenizers as a new and practical vector for membership inference attacks against large language models, addressing evaluation challenges that plague prior MIA work. It introduces five attack methods (three developed here plus two baselines) that exploit overfitting signals in tokenizer training, including Merge Similarity, Vocabulary Overlap, and Frequency Estimation, with an adaptive defense to mitigate leakage. Extensive experiments on millions of real web pages show that tokenizers with larger vocabularies are more vulnerable, and that the Vocabulary Overlap and Frequency Estimation attacks achieve strong discriminative power (high AUC and notable TPR at low FPR). The study highlights an urgent privacy risk and demonstrates that defense mechanisms often trade off tokenizer utility, underscoring the need for privacy-preserving tokenizer design in practical LLM deployments.

Abstract

Membership inference attacks (MIAs) are widely used to assess the privacy risks associated with machine learning models. However, when these attacks are applied to pre-trained large language models (LLMs), they encounter significant challenges, including mislabeled samples, distribution shifts, and discrepancies in model size between experimental and real-world settings. To address these limitations, we introduce tokenizers as a new attack vector for membership inference. Specifically, a tokenizer converts raw text into tokens for LLMs. Unlike full models, tokenizers can be efficiently trained from scratch, thereby avoiding the aforementioned challenges. In addition, the tokenizer's training data is typically representative of the data used to pre-train LLMs. Despite these advantages, the potential of tokenizers as an attack vector remains unexplored. To this end, we present the first study on membership leakage through tokenizers and explore five attack methods to infer dataset membership. Extensive experiments on millions of Internet samples reveal the vulnerabilities in the tokenizers of state-of-the-art LLMs. To mitigate this emerging risk, we further propose an adaptive defense. Our findings highlight tokenizers as an overlooked yet critical privacy threat, underscoring the urgent need for privacy-preserving mechanisms specifically designed for them.

Membership Inference Attacks on Tokenizers of Large Language Models

TL;DR

This work identifies tokenizers as a new and practical vector for membership inference attacks against large language models, addressing evaluation challenges that plague prior MIA work. It introduces five attack methods (three developed here plus two baselines) that exploit overfitting signals in tokenizer training, including Merge Similarity, Vocabulary Overlap, and Frequency Estimation, with an adaptive defense to mitigate leakage. Extensive experiments on millions of real web pages show that tokenizers with larger vocabularies are more vulnerable, and that the Vocabulary Overlap and Frequency Estimation attacks achieve strong discriminative power (high AUC and notable TPR at low FPR). The study highlights an urgent privacy risk and demonstrates that defense mechanisms often trade off tokenizer utility, underscoring the need for privacy-preserving tokenizer design in practical LLM deployments.

Abstract

Membership inference attacks (MIAs) are widely used to assess the privacy risks associated with machine learning models. However, when these attacks are applied to pre-trained large language models (LLMs), they encounter significant challenges, including mislabeled samples, distribution shifts, and discrepancies in model size between experimental and real-world settings. To address these limitations, we introduce tokenizers as a new attack vector for membership inference. Specifically, a tokenizer converts raw text into tokens for LLMs. Unlike full models, tokenizers can be efficiently trained from scratch, thereby avoiding the aforementioned challenges. In addition, the tokenizer's training data is typically representative of the data used to pre-train LLMs. Despite these advantages, the potential of tokenizers as an attack vector remains unexplored. To this end, we present the first study on membership leakage through tokenizers and explore five attack methods to infer dataset membership. Extensive experiments on millions of Internet samples reveal the vulnerabilities in the tokenizers of state-of-the-art LLMs. To mitigate this emerging risk, we further propose an adaptive defense. Our findings highlight tokenizers as an overlooked yet critical privacy threat, underscoring the urgent need for privacy-preserving mechanisms specifically designed for them.

Paper Structure

This paper contains 19 sections, 1 theorem, 16 equations, 15 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Tokenizer Training
  4. Membership Inference
  5. Threat Models
  6. Attack Methodology
  7. MIA via Merge Similarity
  8. Improved MIA via Vocabulary Overlap
  9. Efficient MIA via Frequency Estimation
  10. Attack Evaluation
  11. Experimental Setup
  12. Main Results
  13. Ablation Study
  14. Adaptive Defense
  15. Additional Investigations
  16. ...and 4 more sections

Key Result

Theorem 4.2

Under the power-law distribution clauset2009power, the frequency $\Pr(t_i \mid \mathcal{V}_\text{target})$ of a token $t_i \in \mathcal{V}_\text{target}$ is proportional to $1/i^\alpha$: where $i > x_\textup{min}$, and $\alpha\in\mathbb{R}_{> 0}$ , $x_\textup{min}\in\mathbb{Z}_{> 0}$ are constants defined by the power law. Then, RTF-SI can be approximated by its lower bound:

Figures (15)

  • Figure 1: Evaluation challenges in MIAs against LLMs.
  • Figure 2: Performance of our MIAs on tokenizers of LLMs. Key Finding: Scaling up LLMs mayilvahanan2025llms involves expanding the tokenizer's vocabulary tao2024scalinghuangover and thus improving its compression efficiency (i.e., bytes per token) liu-etal-2025-superbpe. Yet, our figures show that it also increases tokenizer's vulnerability to MIAs.
  • Figure 3: Average merge index for tokens in $\mathcal{V}_\text{in}$ and $\mathcal{V}_\text{out}$. It is shown that overall merge orders in $\mathcal{V}_\text{in}$ and $\mathcal{V}_\text{out}$ resemble.
  • Figure 4: Distinctive tokens in MIA via $\mathsf{Vocabulary~Overlap}$.
  • Figure 5: Relationship between token merge index and frequency in training data, indicating they follow a power law.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Definition 4.1: Relative Token Frequency with Self-information
  • Theorem 4.2: RTF-SI under the Power Law
  • proof