Machine Text Detectors are Membership Inference Attacks

TL;DR

This work reveals a fundamental link between membership inference attacks and machine-generated text detection: both tasks effectively approximate the true distribution of human-written text via a language model. The authors prove that a likelihood-ratio statistic $\Lambda(x)$ is asymptotically optimal for both tasks and show that many existing metrics from either domain can be viewed as approximations to this ratio. Through large-scale experiments across 13 domains and 10 generators, they observe strong cross-task rank correlations (e.g., $\rho>0.6$, rising to $\rho=0.78$ among top methods) and identify Binoculars as a cross-task standout. They also introduce MINT, a unified evaluation suite for fair cross-task benchmarking, and argue for greater collaboration between the two communities to accelerate progress and robust evaluation.

Abstract

Although membership inference attacks (MIAs) and machine-generated text detection target different goals, identifying training samples and synthetic texts, their methods often exploit similar signals based on a language model's probability distribution. Despite this shared methodological foundation, the two tasks have been independently studied, which may lead to conclusions that overlook stronger methods and valuable insights developed in the other task. In this work, we theoretically and empirically investigate the transferability, i.e., how well a method originally developed for one task performs on the other, between MIAs and machine text detection. For our theoretical contribution, we prove that the metric that achieves the asymptotically highest performance on both tasks is the same. We unify a large proportion of the existing literature in the context of this optimal metric and hypothesize that the accuracy with which a given method approximates this metric is directly correlated with its transferability. Our large-scale empirical experiments, including 7 state-of-the-art MIA methods and 5 state-of-the-art machine text detectors across 13 domains and 10 generators, demonstrate very strong rank correlation (rho > 0.6) in cross-task performance. We notably find that Binoculars, originally designed for machine text detection, achieves state-of-the-art performance on MIA benchmarks as well, demonstrating the practical impact of the transferability. Our findings highlight the need for greater cross-task awareness and collaboration between the two research communities. To facilitate cross-task developments and fair evaluations, we introduce MINT, a unified evaluation suite for MIAs and machine-generated text detection, with implementation of 15 recent methods from both tasks.

Figures (3)

• Figure 2: Relationship between method rankings across MIAs and machine text detection. Blue and red plots show MIA methods and machine text detectors. Gray plots indicate general baselines. Dashed line denotes equal ranks.
• Figure 3: AUROC scores for membership inference attacks (blue bars) and generated text detectors (red bars) across both tasks. Top: Membership inference results on the MIMIR benchmark with 13-gram de-duplication filtering averaged over five domains and five models. Middle and Bottom: Generated text detection results on white-box and black-box settings from the RAID benchmark, averaged over eight domains and five models. Both MIA methods (blue bars) and machine text detectors (red bars) have comparable performance in the cross-task setting. Notably, Binoculars achieves the best average performance in both tasks. Full results are provided in Appendix \ref{['full_table_mia_detection']}.
• Figure 4: Zlib compression entropy distribution in MIAs (non-members vs. members) and machine text detection (humans vs. machines), averaged over 3,000 randomly sampled texts for each dataset. The entropy converges between classes in MIAs but diverges in detection with a long tail, due to different prior distributions.