Differential Privacy Personalized Federated Learning Based on Dynamically Sparsified Client Updates

TL;DR

This work addresses the tension between privacy guarantees and personalization in federated learning under data heterogeneity. It introduces DP-pFedDSU, a two-pronged approach combining reparameterization-based sparsified updates with a dynamic adaptive norm to minimize information leakage while preserving performance. The method employs Renyi differential privacy and the moments accountant for privacy accounting, and demonstrates superior accuracy under the same privacy budget on EMNIST, CIFAR-10, and CIFAR-100, with robust performance across data distributions. The findings highlight a practical strategy to enhance privacy-utility tradeoffs in personalized federated learning and point toward scaling to larger models and more complex settings in future work.

Abstract

Personalized federated learning is extensively utilized in scenarios characterized by data heterogeneity, facilitating more efficient and automated local training on data-owning terminals. This includes the automated selection of high-performance model parameters for upload, thereby enhancing the overall training process. However, it entails significant risks of privacy leakage. Existing studies have attempted to mitigate these risks by utilizing differential privacy. Nevertheless, these studies present two major limitations: (1) The integration of differential privacy into personalized federated learning lacks sufficient personalization, leading to the introduction of excessive noise into the model. (2) It fails to adequately control the spatial scope of model update information, resulting in a suboptimal balance between data privacy and model effectiveness in differential privacy federated learning. In this paper, we propose a differentially private personalized federated learning approach that employs dynamically sparsified client updates through reparameterization and adaptive norm(DP-pFedDSU). Reparameterization training effectively selects personalized client update information, thereby reducing the quantity of updates. This approach minimizes the introduction of noise to the greatest extent possible. Additionally, dynamic adaptive norm refers to controlling the norm space of model updates during the training process, mitigating the negative impact of clipping on the update information. These strategies substantially enhance the effective integration of differential privacy and personalized federated learning. Experimental results on EMNIST, CIFAR-10, and CIFAR-100 demonstrate that our proposed scheme achieves superior performance and is well-suited for more complex personalized federated learning scenarios.

Figures (3)

• Figure 1: The overflow of client training in the proposed method.
• Figure 2: Global training loss under different training epochs.
• Figure : Differential Privacy Personalized Federated Learning Based on Dynamically Sparsified Client Updates(DP-pFedDSU)

Theorems & Definitions (4)

• Definition 1
• Definition 2
• Definition 3
• Theorem 1