Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extracting Training Dialogue Data from Large Language Model based Task Bots

Shuo Zhang, Junzhou Zhao, Junji Hou, Pinghui Wang, Chenxu Wang, Jing Tao

TL;DR

This work evaluates existing training data extraction attacks, analyzes key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposes novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference.

Abstract

Large Language Models (LLMs) have been widely adopted to enhance Task-Oriented Dialogue Systems (TODS) by modeling complex language patterns and delivering contextually appropriate responses. However, this integration introduces significant privacy risks, as LLMs, functioning as soft knowledge bases that compress extensive training data into rich knowledge representations, can inadvertently memorize training dialogue data containing not only identifiable information such as phone numbers but also entire dialogue-level events like complete travel schedules. Despite the critical nature of this privacy concern, how LLM memorization is inherited in developing task bots remains unexplored. In this work, we address this gap through a systematic quantitative study that involves evaluating existing training data extraction attacks, analyzing key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposing novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference. Experimental results demonstrate the effectiveness of our proposed data extraction attack. Our method can extract thousands of training labels of dialogue states with best-case precision exceeding 70%. Furthermore, we provide an in-depth analysis of training data memorization in LLM-based TODS by identifying and quantifying key influencing factors and discussing targeted mitigation strategies.

Extracting Training Dialogue Data from Large Language Model based Task Bots

TL;DR

This work evaluates existing training data extraction attacks, analyzes key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposes novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference.

Abstract

Large Language Models (LLMs) have been widely adopted to enhance Task-Oriented Dialogue Systems (TODS) by modeling complex language patterns and delivering contextually appropriate responses. However, this integration introduces significant privacy risks, as LLMs, functioning as soft knowledge bases that compress extensive training data into rich knowledge representations, can inadvertently memorize training dialogue data containing not only identifiable information such as phone numbers but also entire dialogue-level events like complete travel schedules. Despite the critical nature of this privacy concern, how LLM memorization is inherited in developing task bots remains unexplored. In this work, we address this gap through a systematic quantitative study that involves evaluating existing training data extraction attacks, analyzing key characteristics of task-oriented dialogue modeling that render existing methods ineffective, and proposing novel attack techniques tailored for LLM-based TODS that enhance both response sampling and membership inference. Experimental results demonstrate the effectiveness of our proposed data extraction attack. Our method can extract thousands of training labels of dialogue states with best-case precision exceeding 70%. Furthermore, we provide an in-depth analysis of training data memorization in LLM-based TODS by identifying and quantifying key influencing factors and discussing targeted mitigation strategies.
Paper Structure (46 sections, 4 equations, 12 figures, 4 tables)

This paper contains 46 sections, 4 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Work
  3. Large Language Model based Task Bots
  4. Causal Language Modeling for Task-Oriented Dialogues
  5. Training Data Extraction Attacks
  6. Threat Model
  7. Definition of Memorization
  8. Target and Adversary
  9. Adversary's Objective
  10. Adversary Capabilities
  11. Target Model
  12. Dialogue State Extraction Methodology
  13. A Strawman Method for Dialogue State Extraction
  14. Candidate Dialogue State Generation
  15. Membership Inference
  16. ...and 31 more sections

Figures (12)

  • Figure 1: (a) LLM-based task bot utilizes a single neural auto-regressive model to parameterize the sequential dialogue pipeline. (b) Our dialogue state extraction attack extracts the training labels of Dialogue States$(S)$without access to the corresponding conditioning text of dialogue histories.
  • Figure 2: (a) Workflow of the dialogue state extraction attack: We begin with Suffix Decoding, generating many samples from the LLM-based task bot by seeding the model with either empty inputs (for untargeted attack) or partial dialogue state prefixes (for targeted attack). Membership Inference metrics are then applied to rank each generation, with higher ranks indicating a greater likelihood of originating from the training data. For evaluation, we deduplicate generations, keep the top 100, and mark each generation as either memorized or not memorized by comparing them with the dialogue states from the training dataset. (b) Schema-Guided Sampling \ref{['sec:lmvslm']} generates more valid results by constraining word choices for sampling domains and slots using the dialogue schema (i.e., service scope), which is automatically explored by simulating user interactions with ChatGPT. (c) Debiased Conditional Perplexity \ref{['sec:adv_mia']} reduces bias toward favoring common generic dialogue state components.
  • Figure 3: Schema extraction with ChatGPT. In (a) model-against-model loop, based on the collected schema (none for the first loop), ChatGPT is prompted to simulate a user asking the task bot questions to uncover additional domains and slots. To improve precision and exploration, we further propose to enhance the basic loop with (b) inter-temperature consistency. When decoding the answer from the task bot as in (a), we use high-temperature settings to generate diverse belief states, which may include incorrect domains and slots. We then filter these errors through a verification process, where ChatGPT is prompted to ask corresponding questions about the extracted (probably incorrect) states, and the task bot decodes answers under low-temperature settings to generate confident answers that exclude those illegal domains and slots.
  • Figure 4: Schema extraction performance over different temperatures. We report the mean values over three runs per method, along with the best-case results.
  • Figure 5: Minimum number of questions required for ChatGPT to achieve the performance upper bound in schema extraction. For temperature sampling, $10,000$ samples are generated to reach the reported performance.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Example 1: Restaurant Booking Assistant Bot
  • Definition 1: Dialogue State Memorization for Task Bots