Certifying the Right to Be Forgotten: Primal-Dual Optimization for Sample and Label Unlearning in Vertical Federated Learning

TL;DR

This paper tackles the problem of providing data-forgetting guarantees in vertical federated learning (VFL) by introducing FedORA, a primal-dual optimization framework that enables sample and label unlearning. It formulates unlearning as a constrained problem and solves it via a primal-dual hybrid gradient method, employing an uncertainty-based unlearning loss, adaptive step sizes, and an asymmetric batch design to reduce computation and communication. Theoretical analysis yields a bound on the distance between FedORA's unlearned model and a Train-from-scratch retrain, and extensive experiments on tabular and image datasets demonstrate effective forgetting with preserved utility and robust attack resilience. The approach offers practical, scalable unlearning for privacy-sensitive VFL deployments, compatible with privacy-preserving mechanisms and adaptable to varying dataset complexity and unlearning requirements.

Abstract

Federated unlearning has become an attractive approach to address privacy concerns in collaborative machine learning, for situations when sensitive data is remembered by AI models during the machine learning process. It enables the removal of specific data influences from trained models, aligning with the growing emphasis on the "right to be forgotten." While extensively studied in horizontal federated learning, unlearning in vertical federated learning (VFL) remains challenging due to the distributed feature architecture. VFL unlearning includes sample unlearning that removes specific data points' influence and label unlearning that removes entire classes. Since different parties hold complementary features of the same samples, unlearning tasks require cross-party coordination, creating computational overhead and complexities from feature interdependencies. To address such challenges, we propose FedORA (Federated Optimization for data Removal via primal-dual Algorithm), designed for sample and label unlearning in VFL. FedORA formulates the removal of certain samples or labels as a constrained optimization problem solved using a primal-dual framework. Our approach introduces a new unlearning loss function that promotes classification uncertainty rather than misclassification. An adaptive step size enhances stability, while an asymmetric batch design, considering the prior influence of the remaining data on the model, handles unlearning and retained data differently to efficiently reduce computational costs. We provide theoretical analysis proving that the model difference between FedORA and Train-from-scratch is bounded, establishing guarantees for unlearning effectiveness. Experiments on tabular and image datasets demonstrate that FedORA achieves unlearning effectiveness and utility preservation comparable to Train-from-scratch with reduced computation and communication overhead.

Figures (8)

• Figure 1: Difference among sample, label and feature unlearning.
• Figure 2: The VFL framework adopted in the paper.
• Figure 3: Overview of FedORA. Primal and dual variables are updated to achieve unlearning while maintaining model utility.
• Figure 4: Unlearning accuracy for different sample unlearning.
• Figure 5: Unlearning accuracy for different label unlearning.

Theorems & Definitions (1)

• Theorem 1: Model difference upper bound between FedORA and Train-from-scratch