Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks

TL;DR

Forget by Uncertainty introduces Orthogonal Entropy Unlearning (OEU) to enable principled unlearning in quantized neural networks. It combines Entropy-Guided Unlearning, which drives forgotten data toward maximum uncertainty, with Gradient Orthogonal Projection to ensure forgetting updates do not interfere with retained knowledge, backed by first-order guarantees. Empirically, OEU outperforms prior methods across multiple datasets, models, quantization schemes, and forgetting scenarios, while maintaining high utility and offering tight privacy guarantees. The approach provides a practical, efficient pathway toward privacy-preserving unlearning for on-device QNN deployments.

Abstract

The deployment of quantized neural networks on edge devices, combined with privacy regulations like GDPR, creates an urgent need for machine unlearning in quantized models. However, existing methods face critical challenges: they induce forgetting by training models to memorize incorrect labels, conflating forgetting with misremembering, and employ scalar gradient reweighting that cannot resolve directional conflicts between gradients. We propose OEU, a novel Orthogonal Entropy Unlearning framework with two key innovations: 1) Entropy-guided unlearning maximizes prediction uncertainty on forgotten data, achieving genuine forgetting rather than confident misprediction, and 2) Gradient orthogonal projection eliminates interference by projecting forgetting gradients onto the orthogonal complement of retain gradients, providing theoretical guarantees for utility preservation under first-order approximation. Extensive experiments demonstrate that OEU outperforms existing methods in both forgetting effectiveness and retain accuracy.

Figures (5)

• Figure 1: Motivation of OEU. (a) Similar Labels shifts confidence to a wrong class, while our entropy-guided approach achieves genuine uncertainty. (b) Directional conflict between forgetting and retain gradients. (c) Orthogonal projection eliminates this conflict.
• Figure 2: Overview of the proposed OEU framework. The forget set is trained with entropy maximization to achieve uncertainty, while the retain set uses cross-entropy to maintain accuracy. Gradient orthogonal projection resolves conflicts between the two gradients before parameter update.
• Figure 3: Performance gaps of ResNet18 on CIFAR-100. The three rows correspond to full-precision ResNet18, ResNet18 quantized with PACT choi2018pact, and ResNet18 quantized with DSQ gong2019differentiable, respectively. The unlearning scenario is random data forgetting (10%). A shorter bar (smaller gap) indicates performance is closer to that of the Retrained model. The average gap for each method is calculated by dividing the values in the Aggregate Results bar chart by 4.
• Figure 4: Output probability distribution of a randomly selected forget sample from CIFAR-100 on MobileNetV2 (green: ground-truth class; blue: other classes). Left: output distribution of original model . Right: output distribution after applying max-entropy unlearning.
• Figure 5: Runtime comparison of different unlearning methods on MobileNetV2 with CIFAR-100 under 10% random data forgetting. OEU achieves comparable efficiency to other approximate methods while delivering superior unlearning performance.

Theorems & Definitions (19)

• Theorem 4.1: Bias of Label Manipulation
• Theorem 4.2: Unbiasedness of Maximum Entropy
• Theorem 4.3: Retain Preservation
• Theorem 4.4: Forgetting Effectiveness
• Definition 1.1: Random Labels
• Definition 1.2: Similar Labels
• proof : Proof of Theorem \ref{['thm:label_bias']}
• Remark 1.3
• Definition 1.4: Entropy-Guided Unlearning
• proof : Proof of Theorem \ref{['thm:entropy_unbiased']}