Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CE-U: Cross Entropy Unlearning

Bo Yang

TL;DR

This work proposes CE-U (Cross Entropy Unlearning), a loss function for unlearning that addresses fundamental limitations of gradient ascent approaches that suffer from vanishing gradients when model confidence is high and exploding gradients when confidence is low.

Abstract

Large language models memorize sensitive data from their pretraining corpora. In this work, we propose CE-U (Cross Entropy Unlearning), a loss function for unlearning. CE-U addresses fundamental limitations of gradient ascent approaches that suffer from vanishing gradients when model confidence is high and exploding gradients when confidence is low. We also unify standard cross entropy learning and unlearning into a single framework. On the TOFU benchmark for unlearning, CE-U achieves state-of-the-art results on LLaMA2-7B models without using an extra oracle model or additional positive samples. Our analysis reveals that the problematic gradient ascent component also exists in reinforcement learning algorithms like DPO and GRPO. This suggests that applying CE-U approach to reinforcement learning could be promising to improve stability and convergence.

CE-U: Cross Entropy Unlearning

TL;DR

This work proposes CE-U (Cross Entropy Unlearning), a loss function for unlearning that addresses fundamental limitations of gradient ascent approaches that suffer from vanishing gradients when model confidence is high and exploding gradients when confidence is low.

Abstract

Large language models memorize sensitive data from their pretraining corpora. In this work, we propose CE-U (Cross Entropy Unlearning), a loss function for unlearning. CE-U addresses fundamental limitations of gradient ascent approaches that suffer from vanishing gradients when model confidence is high and exploding gradients when confidence is low. We also unify standard cross entropy learning and unlearning into a single framework. On the TOFU benchmark for unlearning, CE-U achieves state-of-the-art results on LLaMA2-7B models without using an extra oracle model or additional positive samples. Our analysis reveals that the problematic gradient ascent component also exists in reinforcement learning algorithms like DPO and GRPO. This suggests that applying CE-U approach to reinforcement learning could be promising to improve stability and convergence.

Paper Structure

This paper contains 23 sections, 19 equations, 2 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology: The CE-U Algorithm
  4. Experimental Setup
  5. Dataset: TOFU
  6. Training Settings
  7. Evaluation Metrics
  8. Experiment Results
  9. Discussion and Future Work
  10. General CE-U
  11. Gradient Behavior Comparison
  12. Incorporating CE-U into Reinforcement Learning Algorithms
  13. Practical Considerations for CE-U
  14. Conclusion
  15. Appendix
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Performance comparison of CE-U versus baseline methods on LLaMA2-7B with 5% forgetting on the TOFU dataset. The dots in each line represent different settings of total epochs for unlearning.
  • Figure 2: Performance comparison of CE-U versus baseline methods on the TOFU dataset. The dots in each line represent different settings of total epochs for unlearning.