On the Evidentiary Limits of Membership Inference for Copyright Auditing

TL;DR

This work investigates whether membership inference attacks can serve as robust, admissible evidence in adversarial copyright disputes over LLM training data. It introduces a judge–prosecutor–accused protocol and a semantic paraphrasing framework, SAGE, guided by a structure-aware SAE to rewrite training data while preserving meaning. Empirical results show MIAs degrade under semantics-preserving paraphrasing, and SAGE/SAGE-R substantially suppress leakage without harming downstream utility, challenging the viability of MIAs as standalone copyright audits. The paper argues for protocol-aware approaches that align technical evidence with legal standards and adversarial realities.

Abstract

As large language models (LLMs) are trained on increasingly opaque corpora, membership inference attacks (MIAs) have been proposed to audit whether copyrighted texts were used during training, despite growing concerns about their reliability under realistic conditions. We ask whether MIAs can serve as admissible evidence in adversarial copyright disputes where an accused model developer may obfuscate training data while preserving semantic content, and formalize this setting through a judge-prosecutor-accused communication protocol. To test robustness under this protocol, we introduce SAGE (Structure-Aware SAE-Guided Extraction), a paraphrasing framework guided by Sparse Autoencoders (SAEs) that rewrites training data to alter lexical structure while preserving semantic content and downstream utility. Our experiments show that state-of-the-art MIAs degrade when models are fine-tuned on SAGE-generated paraphrases, indicating that their signals are not robust to semantics-preserving transformations. While some leakage remains in certain fine-tuning regimes, these results suggest that MIAs are brittle in adversarial settings and insufficient, on their own, as a standalone mechanism for copyright auditing of LLMs.

Figures (4)

• Figure 1: Judge--Prosecutor--Accused protocol and artifact flow. Party-owned artifacts (prosecutor evidence; accused model/data) are color-coded by role, while auditing and obfuscation techniques (membership inference; paraphrasing/SAGE) are shown in neutral gray.
• Figure 2: SAGE qualitative example. Structure-aware dataset generation in SAGE and SAGE-R. The original document (left) is decomposed into structural and narrative sections. Paraphrasing preserves structural sections verbatim while rewriting narrative content. SAGE retains factual anchors and document structure, whereas SAGE-R removes structural sections and replaces factual entities with placeholders (<<FACT_i>>). Green and red markers indicate preserved and removed components, respectively.
• Figure 3: Average performance of MIAs across datasets and methods, averaged over three independently fine-tuned models, one per paraphraser-generated dataset. (a) LoRA fine-tuning. (b) Full fine-tuning. Lower is better. In both regimes, the same qualitative ordering holds: FT $>$ SOFT $>$ SAGE $>$ SAGE-R $>$ PT.
• Figure 4: AUC--WordSim tradeoff: Average MIA AUC ($\downarrow$ means stronger privacy) vs. WordSim ($\downarrow$ indicates greater divergence from original). Each point represents a dataset-level average over paraphraser models; gray segments connect SAGE to SAGE-R within each dataset. Labels indicate SPS.

Theorems & Definitions (4)

• Definition 2.1: Semantic Equivalence
• Definition 2.2: Robustness
• Definition 3.1: Semantic Persistence Score ($\mathrm{SPS}$)
• Definition 3.2: Word Similarity ($\mathrm{WordSim}$)