Private and Federated Stochastic Convex Optimization: Efficient Strategies for Centralized Systems
Roie Reshef, Kfir Y. Levy
TL;DR
This work tackles differential privacy in federated learning under the stochastic convex optimization framework in centralized systems, addressing both trusted and untrusted servers. It builds on the $\mu^2$-SGD methodology by introducing Noisy-$\mu^2$-SGD and a parallel minibatch FL template that preserves linear computational complexity in the total data size while enabling DP. The authors derive optimal population-loss convergence bounds for both untrusted ($O\left(\frac{1}{\sqrt{nM}}+\frac{\sqrt{d}}{\epsilon n \sqrt{M}}\right)$) and trusted ($O\left(\frac{1}{\sqrt{nM}}+\frac{\sqrt{d}}{\epsilon n M}\right)$) server scenarios, along with gradient-error and sensitivity analyses, and provide empirical validation on MNIST. The approach significantly improves the practicality of DP in FL by balancing privacy, efficiency, and robustness across server-trust environments.
Abstract
This paper addresses the challenge of preserving privacy in Federated Learning (FL) within centralized systems, focusing on both trusted and untrusted server scenarios. We analyze this setting within the Stochastic Convex Optimization (SCO) framework, and devise methods that ensure Differential Privacy (DP) while maintaining optimal convergence rates for homogeneous and heterogeneous data distributions. Our approach, based on a recent stochastic optimization technique, offers linear computational complexity, comparable to non-private FL methods, and reduced gradient obfuscation. This work enhances the practicality of DP in FL, balancing privacy, efficiency, and robustness in a variety of server trust environment.