GRPO Privacy Is at Risk: A Membership Inference Attack Against Reinforcement Learning With Verifiable Rewards

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR) shifts training away from memorized ground-truth outputs, creating privacy leakage channels tied to behavioral changes rather than output memorization. The paper introduces Divergence-in-Behavior Attack (DIBA), a two-axis MIA framework that captures advantage-based correctness gains and logit-side policy divergence (via $k3kl\_approx$) to detect whether a prompt was used in RLVR fine-tuning, reporting up to $AUC \approx 0.84$ and markedly higher $TPR@0.1\%FPR$ than baselines. DIBA generalizes across RLVR variants (GRPO, DAPO) and models (Qwen 3B/7B), extends to vision-language models with competitive AUC, and remains reasonably robust under moderate defenses. The work highlights a practical privacy risk in RLVR, guides the development of privacy auditing tools, and motivates future defenses to mitigate behavioral leakage in on-policy RL frameworks.

Abstract

Membership inference attacks (MIAs) on large language models (LLMs) pose significant privacy risks across various stages of model training. Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have brought a profound paradigm shift in LLM training, particularly for complex reasoning tasks. However, the on-policy nature of RLVR introduces a unique privacy leakage pattern: since training relies on self-generated responses without fixed ground-truth outputs, membership inference must now determine whether a given prompt (independent of any specific response) is used during fine-tuning. This creates a threat where leakage arises not from answer memorization. To audit this novel privacy risk, we propose Divergence-in-Behavior Attack (DIBA), the first membership inference framework specifically designed for RLVR. DIBA shifts the focus from memorization to behavioral change, leveraging measurable shifts in model behavior across two axes: advantage-side improvement (e.g., correctness gain) and logit-side divergence (e.g., policy drift). Through comprehensive evaluations, we demonstrate that DIBA significantly outperforms existing baselines, achieving around 0.8 AUC and an order-of-magnitude higher TPR@0.1%FPR. We validate DIBA's superiority across multiple settings--including in-distribution, cross-dataset, cross-algorithm, black-box scenarios, and extensions to vision-language models. Furthermore, our attack remains robust under moderate defensive measures. To the best of our knowledge, this is the first work to systematically analyze privacy vulnerabilities in RLVR, revealing that even in the absence of explicit supervision, training data exposure can be reliably inferred through behavioral traces.

Figures (20)

• Figure 1: MIAs in different LLM training paradigms.
• Figure 2: Overview of DIBA's feature construction.
• Figure 3: AUC and log-scale ROC curves for DIBA on different models and training algorithms.
• Figure 4: Performance Over Steps: Train reward represents the accuracy on sampled responses from the member prompts, while test reward represents the sampled responses from the non-member prompts.
• Figure 5: Pareto Front Visualization of Feature Performance: pAUC and TPR@FPR$\leq$0.1% are on the axes. Marker size reflects the KS statistic (larger = higher), and black rings indicate Pareto-optimal features.

Theorems & Definitions (1)

• Definition 1