Controllable Unlearning for Image-to-Image Generative Models via $\varepsilon$-Constrained Optimization

TL;DR

The paper tackles privacy- and bias-related concerns in Image-to-Image (I2I) generative models by proposing a controllable unlearning framework that treats unlearning as an $\varepsilon$-constrained optimization problem. It reformulates the problem as bi-objective (unlearning on the forget set vs. retaining utility on the retain set) and derives two boundary Pareto-optimal solutions, guaranteeing a valid range for the control parameter $\varepsilon$ where Pareto optimality holds. A gradient-based solver with a phase-based two-step process identifies unlearning boundaries (Phase I) and then traces a Pareto path by relaxing the constraint (Phase II), with a control function $\psi(\theta)$ guiding convergence. The method is validated across MAE, VQ-GAN, and diffusion I2I models on ImageNet-1K and Places-365, achieving superior forgetting on the forget set while preserving retain-set performance, and enabling fine-grained control over the unlearning-utility trade-off. These results demonstrate a scalable, theoretically grounded approach to customizable unlearning in powerful I2I systems, with potential extensions to other generative domains.

Abstract

While generative models have made significant advancements in recent years, they also raise concerns such as privacy breaches and biases. Machine unlearning has emerged as a viable solution, aiming to remove specific training data, e.g., containing private information and bias, from models. In this paper, we study the machine unlearning problem in Image-to-Image (I2I) generative models. Previous studies mainly treat it as a single objective optimization problem, offering a solitary solution, thereby neglecting the varied user expectations towards the trade-off between complete unlearning and model utility. To address this issue, we propose a controllable unlearning framework that uses a control coefficient $\varepsilon$ to control the trade-off. We reformulate the I2I generative model unlearning problem into a $\varepsilon$-constrained optimization problem and solve it with a gradient-based method to find optimal solutions for unlearning boundaries. These boundaries define the valid range for the control coefficient. Within this range, every yielded solution is theoretically guaranteed with Pareto optimality. We also analyze the convergence rate of our framework under various control functions. Extensive experiments on two benchmark datasets across three mainstream I2I models demonstrate the effectiveness of our controllable unlearning framework.

Figures (24)

• Figure 1: An overview of controllable unlearning. On the left, the first and second rows represent the forget set and the retain set, respectively. We first present the effect of unlearning in I2I generative models, followed by a collection of controllable solutions, where $\varepsilon$ is the control coefficient. On the right, we demonstrate that for each $\varepsilon$, our solution is guaranteed with the Pareto optimality.
• Figure 2: Pipeline of the controllable unlearning framework. (a) shows the unlearning task of the I2I generative model which is framed as a $\varepsilon$-constrained optimization problem. (b) shows that the implementation of controllable unlearning unfolds in two phases: i) initially identifying two boundary points of unlearning, necessitating a strict reduction in $f_1(\theta)$ (or $f_2(\theta)$) for optimality; and ii) then locating the given $\varepsilon$'s Pareto optimal point, with strict reduction in $f_1(\theta)$ when $f_1(\theta_t) > \varepsilon$ and permitting an increase when $f_1(\theta_t) \leq \varepsilon$.
• Figure 3: Generated images of cropping 50% at the center of the image on VQ-GAN. From left to right, the images generated by baselines are presented. Our method results in the highest degree of unlearning completeness while maintaining a minimal reduction in model utility.
• Figure 4: T-SNE analysis between images generated by our method and ground truth images.
• Figure 5: VQ-GAN: generated images of cropping 50% at the center of the image. The upper part (a) represents the forget set, while the lower part (b) represents the retain set. "Ours" denotes the boundary condition of unlearning obtained in Phase I, which represents the point of the highest degree of unlearning completeness. It is evident that our method significantly outperforms baselines in terms of the unlearning effect on the forget set, most closely approximating Gaussian noise, and exhibits the least performance degradation on the retain set.

Theorems & Definitions (13)

• Lemma 1: Divergence Upper Bound cover2012elements
• Proposition 1: Boundary of Pareto Set
• proof
• Proposition 2: Interior of Paret Set
• proof
• Lemma 2
• Lemma 3
• proof : Proof of Proposition \ref{['prop:1']}
• Lemma 4
• Lemma 5