EvoMU: Evolutionary Machine Unlearning

TL;DR

EvoMU tackles the problem of selective forgetting in LLMs by automatically discovering task-specific unlearning losses through an evolutionary loop guided by a code-capable LLM. The approach uses a small model (e.g., $4$B parameters) to propose, train with LoRA adapters, and mutate candidate losses, achieving SotA forgetting-utility trade-offs on benchmarks such as TOFU, MUSE, and WMDP. By explicitly optimizing the balance between reducing the likelihood of forget data and preserving utility on retain data, EvoMU demonstrates that loss structure, not just hyperparameters, drives performance, and that automated loss design can rival or surpass human-designed formulations with limited compute. The work highlights the potential of automated scientific discovery for practical ML safety tasks, while emphasizing evaluation rigor and responsible use given potential dual-use risks.

Abstract

Machine unlearning aims to unlearn specified training data (e.g. sensitive or copyrighted material). A prominent approach is to fine-tune an existing model with an unlearning loss that retains overall utility. The space of suitable unlearning loss functions is vast, making the search for an optimal loss function daunting. Additionally, there might not even exist a universally optimal loss function: differences in the structure and overlap of the forget and retain data can cause a loss to work well in one setting but over-unlearn or under-unlearn in another. Our approach EvoMU tackles these two challenges simultaneously. An evolutionary search procedure automatically finds task-specific losses in the vast space of possible unlearning loss functions. This allows us to find dataset-specific losses that match or outperform existing losses from the literature, without the need for a human-in-the-loop. This work is therefore an instance of automatic scientific discovery, a.k.a. an AI co-scientist. In contrast to previous AI co-scientist works, we do so on a budget: We achieve SotA results using a small 4B parameter model (Qwen3-4B-Thinking), showing the potential of AI co-scientists with limited computational resources. Our experimental evaluation shows that we surpass previous loss-based unlearning formulations on TOFU-5%, TOFU-10%, MUSE and WMDP by synthesizing novel unlearning losses. Our code is available at https://github.com/Batorskq/EvoMU.

Figures (3)

• Figure 1: High level overview of EvoMU. A LLM proposes candidate unlearning loss functions along with their training budget. For each candidate, we fine-tune the base model with LoRA adapters on forget and retain data, then evaluate the resulting checkpoint using standard forgetting and utility metrics. We select the top-$K$ objectives and summarize their full feedback (loss code, training curves, and evaluation scores). Conditioned on this feedback, the LLM mutates the objectives by adjusting coefficients, modifying structure, and optionally changing the training budget, to produce the next generation. Repeating this verify--select--mutate loop yields task-specific unlearning losses that match or outperform human-designed ones.
• Figure 2: EvoMU overview. We use an LLM Proposer to generate an initial set of unlearning losses. Each loss is evaluated by training an LLM with it. The top-$K$ candidates are retained and mutated based on the obtained results. The mutation loop is repeated $R$ times.
• Figure :