Universally Harmonizing Differential Privacy Mechanisms for Federated Learning: Boosting Accuracy and Convergence

TL;DR

This paper tackles the privacy-utility trade-off in federated learning by introducing UDP-FL, a universal DP-FL framework that harmonizes multiple DP mechanisms under Rényi DP through a centralized Harmonizer. It integrates mode connectivity-based convergence analysis and augments privacy guarantees with a Shuffler, achieving tighter privacy bounds and faster convergence than state-of-the-art baselines. The Staircase mechanism emerges as particularly effective within UDP-FL, delivering superior accuracy under realistic privacy budgets while maintaining robustness against membership inference, data reconstruction, and attribute inference attacks. Overall, UDP-FL provides a flexible, scalable approach to differentially private federated learning with practical implications for secure, efficient collaborative training.

Abstract

Differentially private federated learning (DP-FL) is a promising technique for collaborative model training while ensuring provable privacy for clients. However, optimizing the tradeoff between privacy and accuracy remains a critical challenge. To our best knowledge, we propose the first DP-FL framework (namely UDP-FL), which universally harmonizes any randomization mechanism (e.g., an optimal one) with the Gaussian Moments Accountant (viz. DP-SGD) to significantly boost accuracy and convergence. Specifically, UDP-FL demonstrates enhanced model performance by mitigating the reliance on Gaussian noise. The key mediator variable in this transformation is the Rényi Differential Privacy notion, which is carefully used to harmonize privacy budgets. We also propose an innovative method to theoretically analyze the convergence for DP-FL (including our UDP-FL ) based on mode connectivity analysis. Moreover, we evaluate our UDP-FL through extensive experiments benchmarked against state-of-the-art (SOTA) methods, demonstrating superior performance on both privacy guarantees and model performance. Notably, UDP-FL exhibits substantial resilience against different inference attacks, indicating a significant advance in safeguarding sensitive data in federated learning environments.

Figures (6)

• Figure 1: Accuracy and convergence results of UDP-FL and the baselines. 1) among the three mechanisms, the Staircase always performs the best with the same privacy budget; 2) UDP-FL obtains significantly better privacy-utility tradeoff and faster convergence than the baseline; and 3) UDP-FL (Staircase) even has a comparable accuracy with FedAvg (No DP).
• Figure 2: UDP-FL on CIFAR-10 when (a) $\epsilon = 2$ and (b) $\epsilon = 8$. For a small privacy budget ($\epsilon = 2$), DP-SGD yields better performance, while at a larger privacy budget ($\epsilon = 8$), UDP-FL with Staircase mechanism outperforms DP-SGD and Laplace.
• Figure 3: Impact of the number of clients and sampling rate on UDP-FL. We observe that: 1) when the number of clients increases, shown in Figures (a) and (c), UDP-FL needs more epochs to converge; 2) with the increase of data sampling rate shown in Figures (b) and (d), UDP-FL converges faster.
• Figure 4: Performance evaluation of UDP-FL on MNIST and Medical datasets. (a) and (c) present UDP-FL's accuracy under various differential privacy noise mechanisms, compared to a non-private baseline, with varying $\epsilon$ values. (b) and (d) illustrate the learning curves over training epochs for the MNIST and Medical datasets without privacy and with DP guarantees.
• Figure 5: Accuracy results on FL with the Scaffold aggregation karimireddy2020scaffold. On small training epochs, UDP-FL with the Staircase mechanism can achieve better accuracy more quickly.

Theorems & Definitions (16)

• Definition 1: $(\epsilon,\delta)$-Differential Privacy dwork2006calibratingdwork2006differential
• Definition 2: $(\alpha,\gamma)$-Rényi Differential Privacy mironov2017renyi
• Theorem 1: Proof in Appendix \ref{['proof:stairrdprivacy']}
• Lemma 1: Proof in Geng et al. staircase
• Theorem 2: Proof in Appendix \ref{['prv:stairrivergence']}
• Theorem 3: Proof in Appendix \ref{['proof:staircaseu']}
• Theorem 4: Proof in Appendix \ref{['proof:mode2dp']}
• Theorem 5
• Lemma 2
• Lemma 3