Adaptive Clipping for Privacy-Preserving Few-Shot Learning: Enhancing Generalization with Limited Data
Kanishka Ranaweera, Dinh C. Nguyen, Pubudu N. Pathirana, David Smith, Ming Ding, Thierry Rakotoarivelo, Aruna Seneviratne
TL;DR
This work tackles privacy-preserving few-shot learning by introducing Meta-Clip, an adaptive gradient clipping mechanism that augments differential privacy (DP) in meta-learning. Meta-Clip is integrated into three prominent DP meta-learning algorithms—DP-MAML, DP-Reptile, and DP-MetaSGD—accompanied by a privacy analysis based on Rényi differential privacy and a convergence study to a $\mu$-first-order stationary point. The authors provide extensive empirical evidence on Omniglot and Mini-ImageNet, showing consistent privacy-utility gains over baselines across 1-shot and 5-shot tasks and multiple privacy budgets $\varepsilon$. The approach enables secure, accurate meta-learners under limited data, broadening the applicability of privacy-preserving models in real-world, data-constrained settings.
Abstract
In the era of data-driven machine-learning applications, privacy concerns and the scarcity of labeled data have become paramount challenges. These challenges are particularly pronounced in the domain of few-shot learning, where the ability to learn from limited labeled data is crucial. Privacy-preserving few-shot learning algorithms have emerged as a promising solution to address such pronounced challenges. However, it is well-known that privacy-preserving techniques often lead to a drop in utility due to the fundamental trade-off between data privacy and model performance. To enhance the utility of privacy-preserving few-shot learning methods, we introduce a novel approach called Meta-Clip. This technique is specifically designed for meta-learning algorithms, including Differentially Private (DP) model-agnostic meta-learning, DP-Reptile, and DP-MetaSGD algorithms, with the objective of balancing data privacy preservation with learning capacity maximization. By dynamically adjusting clipping thresholds during the training process, our Adaptive Clipping method provides fine-grained control over the disclosure of sensitive information, mitigating overfitting on small datasets and significantly improving the generalization performance of meta-learning models. Through comprehensive experiments on diverse benchmark datasets, we demonstrate the effectiveness of our approach in minimizing utility degradation, showcasing a superior privacy-utility trade-off compared to existing privacy-preserving techniques. The adoption of Adaptive Clipping represents a substantial step forward in the field of privacy-preserving few-shot learning, empowering the development of secure and accurate models for real-world applications, especially in scenarios where there are limited data availability.