Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DMRL: Data- and Model-aware Reward Learning for Data Extraction

Zhiqiang Wang, Ruoxi Cheng

TL;DR

DMRL tackles privacy leakage in LLMs by framing data extraction as an inverse reinforcement learning problem, learning shadow reward models from a curated privacy-leakage Q&A dataset. It integrates data- and model-hardness signals through DMHM to adapt policy updates, using GRPO-S to stabilize and scale learning. Empirical results demonstrate that DMRL outperforms baselines in PII extraction, reconstruction, and inference across multiple models and datasets, highlighting the method's robustness and potential for red-teaming and defense design. The work advances privacy-safety research for LLMs by offering a scalable, hardness-aware framework that couples IRL-based reward modeling with distribution-aware optimization, though it emphasizes responsible usage and risk mitigation in deployment.

Abstract

Large language models (LLMs) are inherently vulnerable to unintended privacy breaches. Consequently, systematic red-teaming research is essential for developing robust defense mechanisms. However, current data extraction methods suffer from several limitations: (1) rely on dataset duplicates (addressable via deduplication), (2) depend on prompt engineering (now countered by detection and defense), and (3) rely on random-search adversarial generation. To address these challenges, we propose DMRL, a Data- and Model-aware Reward Learning approach for data extraction. This technique leverages inverse reinforcement learning to extract sensitive data from LLMs. Our method consists of two main components: (1) constructing an introspective reasoning dataset that captures leakage mindsets to guide model behavior, and (2) training reward models with Group Relative Policy Optimization (GRPO), dynamically tuning optimization based on task difficulty at both the data and model levels. Comprehensive experiments across various LLMs demonstrate that DMRL outperforms all baseline methods in data extraction performance.

DMRL: Data- and Model-aware Reward Learning for Data Extraction

TL;DR

DMRL tackles privacy leakage in LLMs by framing data extraction as an inverse reinforcement learning problem, learning shadow reward models from a curated privacy-leakage Q&A dataset. It integrates data- and model-hardness signals through DMHM to adapt policy updates, using GRPO-S to stabilize and scale learning. Empirical results demonstrate that DMRL outperforms baselines in PII extraction, reconstruction, and inference across multiple models and datasets, highlighting the method's robustness and potential for red-teaming and defense design. The work advances privacy-safety research for LLMs by offering a scalable, hardness-aware framework that couples IRL-based reward modeling with distribution-aware optimization, though it emphasizes responsible usage and risk mitigation in deployment.

Abstract

Large language models (LLMs) are inherently vulnerable to unintended privacy breaches. Consequently, systematic red-teaming research is essential for developing robust defense mechanisms. However, current data extraction methods suffer from several limitations: (1) rely on dataset duplicates (addressable via deduplication), (2) depend on prompt engineering (now countered by detection and defense), and (3) rely on random-search adversarial generation. To address these challenges, we propose DMRL, a Data- and Model-aware Reward Learning approach for data extraction. This technique leverages inverse reinforcement learning to extract sensitive data from LLMs. Our method consists of two main components: (1) constructing an introspective reasoning dataset that captures leakage mindsets to guide model behavior, and (2) training reward models with Group Relative Policy Optimization (GRPO), dynamically tuning optimization based on task difficulty at both the data and model levels. Comprehensive experiments across various LLMs demonstrate that DMRL outperforms all baseline methods in data extraction performance.
Paper Structure (28 sections, 21 equations, 2 figures, 3 tables, 2 algorithms)

This paper contains 28 sections, 21 equations, 2 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Extraction in LLMs
  4. Reward Learning
  5. Methodology
  6. Privacy-leakage Dataset
  7. Shadow Reward Models Using IRL
  8. Reward Learning via IRL
  9. Algorithm Interpretation
  10. Data and Model Hardness Measurement
  11. Text Splitting
  12. Data Hardness Measurement
  13. Model Hardness Measurement
  14. Hardness Combination
  15. GRPO-Scaling
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Pipeline of the proposed method. First, we generate a Q&A privacy-leakage dataset using structured prompts to serve as demonstration data for training the reward model. Next, we leverage this reward model within GRPO to fine-tune the LLM, dynamically adjusting optimization according to task difficulty at both data and model levels.
  • Figure 2: PII reconstruction results across different LLMs.