Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

TL;DR

This work tackles the challenge of unlearning factual knowledge in large language models by proposing Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set and uses in-distribution alternate labels as positive guidance. It introduces the TOFU benchmark and two new metrics, Forget Utility (FU) and Cleanness Indistinguishability (CI), to robustly evaluate forgetting quality and privacy leakage alongside existing FQ/MU metrics. Through extensive experiments on TOFU with Llama2/Llama3.2, AltPO achieves superior forgetting while preserving utility and displaying stable training, outperforming prior approaches that relied solely on negative feedback. The results provide practical guidance for safer, more reliable unlearning in LLMs and highlight the importance of in-domain positive feedback and multiple alternate labels for robust forgetting.

Abstract

Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance. Our implementation can be found at https://github.com/molereddy/Alternate-Preference-Optimization.

Figures (21)

• Figure 1: The unlearning pipeline and the resulting generations post unlearning with different methods.
• Figure 2: The AltPO unlearning algorithm
• Figure 3: Trajectory of MU versus log(FQ) for different unlearning methods. Marker size represents the epoch number. Trajectories are reported for the 10, 5, 1% splits of TOFU in order, on Llama2.
• Figure 4: Trajectory of FU throughout the unlearning process for $10\%$ forget split of TOFU, using Llama2.
• Figure 5: Trajectory of Cleanness Indistinguishability (CI) throughout the unlearning process. Trajectories are reported for the 10% split of TOFU on Llama2.