Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection

Ali Naseh, Niloofar Mireshghallah

TL;DR

This paper investigates the reliability of membership inference attacks (MIAs) for evaluating memorization in large language models when synthetic data are used for non-member sets. It shows that MIAs effectively act as detectors of machine-generated text, misclassifying synthetic continuations as training members across architectures, which can drive AUC values below random in many cases. The study also links MIAs to zero-shot machine-generated text detectors, arguing that both rely on similar probability-surface signals, which synthetic text naturally occupies. The results imply that synthetic-data–driven evaluation protocols can yield false conclusions about memorization or data leakage and motivate approaches like compression-based normalization and redesigned non-member selection to restore robustness.

Abstract

Recent work shows membership inference attacks (MIAs) on large language models (LLMs) produce inconclusive results, partly due to difficulties in creating non-member datasets without temporal shifts. While researchers have turned to synthetic data as an alternative, we show this approach can be fundamentally misleading. Our experiments indicate that MIAs function as machine-generated text detectors, incorrectly identifying synthetic data as training samples regardless of the data source. This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5. Even synthetic text generated by different, potentially larger models is classified as training data by the target model. Our findings highlight a serious concern: using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage. We caution that this issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.

Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection

TL;DR

This paper investigates the reliability of membership inference attacks (MIAs) for evaluating memorization in large language models when synthetic data are used for non-member sets. It shows that MIAs effectively act as detectors of machine-generated text, misclassifying synthetic continuations as training members across architectures, which can drive AUC values below random in many cases. The study also links MIAs to zero-shot machine-generated text detectors, arguing that both rely on similar probability-surface signals, which synthetic text naturally occupies. The results imply that synthetic-data–driven evaluation protocols can yield false conclusions about memorization or data leakage and motivate approaches like compression-based normalization and redesigned non-member selection to restore robustness.

Abstract

Recent work shows membership inference attacks (MIAs) on large language models (LLMs) produce inconclusive results, partly due to difficulties in creating non-member datasets without temporal shifts. While researchers have turned to synthetic data as an alternative, we show this approach can be fundamentally misleading. Our experiments indicate that MIAs function as machine-generated text detectors, incorrectly identifying synthetic data as training samples regardless of the data source. This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5. Even synthetic text generated by different, potentially larger models is classified as training data by the target model. Our findings highlight a serious concern: using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage. We caution that this issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.
Paper Structure (18 sections, 1 figure, 1 table)

This paper contains 18 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Membership Inference Attacks and Generated Text Detection
  3. Membership Inference Attacks (MIAs)
  4. Zero-Shot Machine-Generated Text Detection
  5. Why MIAs Can Function as Machine-generated Text Detectors
  6. Experimental Setup
  7. Data
  8. Target Model and Attacks
  9. Evaluation Protocol
  10. Experimental Results
  11. Misclassification of Synthetic Text as Training Data.
  12. Cross-Model Transfer.
  13. Implications for Evaluation Protocols.
  14. Zlib as an Outlier.
  15. Discussion and Future Work
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Overview of our methodology: The conventional setup (top) evaluates MIAs using human-written members and non-members from MIMIR, while the synthetic setup (bottom) replaces non-members with machine-generated continuations, produced by prompting generator models with the first 30 tokens of each non-member. The AUC drops from 0.66 to 0.20 between setups, with AUC $\ll$ 0.5 indicating that MIAs consistently misclassify synthetic text as training data. Both setups use GPT-Neo 2.7B as the target model.