Measuring memorization in RLHF for code completion

TL;DR

This work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models, and is robust across multiple code completion datasets, tasks, and model scales.

Abstract

Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences. Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment process. Understanding this relationship is important as real user data may be collected and used to align large models; if user data is memorized during RLHF and later regurgitated, this could raise privacy concerns. In addition to RLHF, other methods such as Direct Preference Optimization (DPO) and $Ψ$PO have gained popularity for learning directly from human preferences, removing the need for optimizing intermediary reward models with reinforcement learning. In this work, we analyze how training data memorization can surface and propagate through each phase of RLHF and direct preference learning. We focus our study on code completion models, as code completion is one of the most popular use cases for large language models. We find that RLHF significantly decreases the chance that data used for reward modeling and reinforcement learning is memorized in comparison to directly fine-tuning on this data, but that examples already memorized during the fine-tuning stage of RLHF, will, in the majority of cases, remain memorized after RLHF. In contrast, we find that aligning by learning directly from human preference data via a special case of $Ψ$PO, Identity Preference Optimization (IPO), increases the likelihood that training data is regurgitated compared to RLHF. Our work suggests that RLHF, as opposed to direct preference learning, is a safer way to mitigate the risk of regurgitating sensitive preference data when aligning large language models. We find our conclusions are robust across multiple code completion datasets, tasks, and model scales.

Figures (9)

• Figure 1: Comparison of memorization rates on fine-tuning training data before (FT.1) and after RL fine-tuning (RLFT.1). We obtain examples on SD.Base that are memorized after fine-tuning, and then record the distance between the target and completion after RL fine-tuning, where we vary the KL penalty $\alpha$. A small $\alpha$ allows the RL fine-tuned model to deviate from the initial FT model, decreasing its memorization of FT training data.
• Figure 2: Distribution of reward scores on SD.Links across reward models. The reward models trained on SD clearly separate negative and positive examples, showing they fit the data well (as required to examine potential memorization of SD by the downstream RLFT model).
• Figure 3: Normalized edit distance between model completions and targets on SD.Base. In \ref{['fig: ft_exp_3', 'fig: rlhf_exp_3']}, we compare edit distance between a fine-tuned model and RL fine-tuned model that included SD.Base in its training data, respectively, and a fine-tuned (control) model that did not include SD.Base in its training data. We highlight areas where examples have small edit distances with respect to the control model; these are potential false positives of memorization, and where examples have small edit distances with respect to the model under inspection but not under the control model; these are likely to be true positives of memorization. Overall, there are very few cases of memorization in RLFT.
• Figure 4: Memorization rates of RL fine-tuning prompts for varying levels of KL regularization. We see an increase in memorization of RLFT data as the RLFT.4 model is allowed to deviate further from its initialization (lower $\alpha$). However, still $<0.5\%$ of training prompts are memorized after 70 epochs.
• Figure 5: Memorization rates of RL fine-tuning prompts for varying levels of KL regularization and learning rate (LR) after 14 epochs. We see an increase in memorization of RLFT data as the RLFT.4 model is allowed to deviate further from its initialization (lower $\alpha$ and larger LR).

Theorems & Definitions (1)

• Definition 3.1: Counterfactual memorization