Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Hard Tokens Reveal: Exploiting Low-confidence Tokens for Membership Inference Attacks against Large Language Models

Md Tasnim Jawad, Mingyan Xiao, Yanzhao Wu

TL;DR

HT-MIA tackles privacy risks in fine-tuned LLMs by shifting from sequence-level signals to token-level analysis of hard tokens. It compares token probabilities of a target model and a reference model, selecting the K hardest positions to compute a robust membership score that is thresholded to detect training data membership. Across medical and general-domain datasets, HT-MIA consistently outperforms seven baselines, with strong gains at strict false-positive rates and in larger models, while DP-SGD provides a practical defense by reducing attack efficacy at the cost of some utility. The work demonstrates the value of fine-grained, cross-model, token-level analysis for strengthening privacy auditing and guiding defense strategies in LLM deployments.

Abstract

With the widespread adoption of Large Language Models (LLMs) and increasingly stringent privacy regulations, protecting data privacy in LLMs has become essential, especially for privacy-sensitive applications. Membership Inference Attacks (MIAs) attempt to determine whether a specific data sample was included in the model training/fine-tuning dataset, posing serious privacy risks. However, most existing MIA techniques against LLMs rely on sequence-level aggregated prediction statistics, which fail to distinguish prediction improvements caused by generalization from those caused by memorization, leading to low attack effectiveness. To address this limitation, we propose a novel membership inference approach that captures the token-level probabilities for low-confidence (hard) tokens, where membership signals are more pronounced. By comparing token-level probability improvements at hard tokens between a fine-tuned target model and a pre-trained reference model, HT-MIA isolates strong and robust membership signals that are obscured by prior MIA approaches. Extensive experiments on both domain-specific medical datasets and general-purpose benchmarks demonstrate that HT-MIA consistently outperforms seven state-of-the-art MIA baselines. We further investigate differentially private training as an effective defense mechanism against MIAs in LLMs. Overall, our HT-MIA framework establishes hard-token based analysis as a state-of-the-art foundation for advancing membership inference attacks and defenses for LLMs.

What Hard Tokens Reveal: Exploiting Low-confidence Tokens for Membership Inference Attacks against Large Language Models

TL;DR

HT-MIA tackles privacy risks in fine-tuned LLMs by shifting from sequence-level signals to token-level analysis of hard tokens. It compares token probabilities of a target model and a reference model, selecting the K hardest positions to compute a robust membership score that is thresholded to detect training data membership. Across medical and general-domain datasets, HT-MIA consistently outperforms seven baselines, with strong gains at strict false-positive rates and in larger models, while DP-SGD provides a practical defense by reducing attack efficacy at the cost of some utility. The work demonstrates the value of fine-grained, cross-model, token-level analysis for strengthening privacy auditing and guiding defense strategies in LLM deployments.

Abstract

With the widespread adoption of Large Language Models (LLMs) and increasingly stringent privacy regulations, protecting data privacy in LLMs has become essential, especially for privacy-sensitive applications. Membership Inference Attacks (MIAs) attempt to determine whether a specific data sample was included in the model training/fine-tuning dataset, posing serious privacy risks. However, most existing MIA techniques against LLMs rely on sequence-level aggregated prediction statistics, which fail to distinguish prediction improvements caused by generalization from those caused by memorization, leading to low attack effectiveness. To address this limitation, we propose a novel membership inference approach that captures the token-level probabilities for low-confidence (hard) tokens, where membership signals are more pronounced. By comparing token-level probability improvements at hard tokens between a fine-tuned target model and a pre-trained reference model, HT-MIA isolates strong and robust membership signals that are obscured by prior MIA approaches. Extensive experiments on both domain-specific medical datasets and general-purpose benchmarks demonstrate that HT-MIA consistently outperforms seven state-of-the-art MIA baselines. We further investigate differentially private training as an effective defense mechanism against MIAs in LLMs. Overall, our HT-MIA framework establishes hard-token based analysis as a state-of-the-art foundation for advancing membership inference attacks and defenses for LLMs.
Paper Structure (36 sections, 4 theorems, 22 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 36 sections, 4 theorems, 22 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Target Model:
  4. Threat Model:
  5. HT-MIA Overview
  6. Attack Method
  7. Defense Method
  8. Experimental Analysis
  9. Target Model Configurations:
  10. Datasets:
  11. Baselines:
  12. Overall Performance
  13. Medical Dataset
  14. Asclepius dataset:
  15. Clinicalnotes dataset:
  16. ...and 21 more sections

Key Result

Lemma 1

Let $\mu_i := \mathop{\mathrm{\mathbb{E}}}\nolimits[\Delta_i \mid x \in D_{\text{mem}}] - \mathop{\mathrm{\mathbb{E}}}\nolimits[\Delta_i \mid x \in D_{\text{non}}]$ be the member–non-member mean gap at token $i$. Assume that $|\mu_i|$ is non-increasing in $p_{\mathcal{F},i}$, so that harder tokens u

Figures (5)

  • Figure 1: Overall Workflow of HT-MIA.
  • Figure 2: Comparison of multiple MIA methods on target models fine-tuned with Wikipedia and IMDB datasets.
  • Figure 3: Token-level probability improvement across target and reference model (Clinicalnotes).
  • Figure 4: AUC comparison of MIA methods on GPT-2 and Qwen-3-0.6B fine-tuned with and without DP-SGD on the Asclepius dataset.
  • Figure 5: Token-level probability improvement across target and reference model (IMDB).

Theorems & Definitions (4)

  • Lemma 1: Informative-token focus
  • Lemma 2: Neyman-Pearson optimality
  • Theorem 1: Finite-sample detection guarantees
  • Corollary 1: Sample complexity for target power