Characterizing the Accuracy-Communication-Privacy Trade-off in Distributed Stochastic Convex Optimization
Sudeep Salgia, Nikola Pavlovic, Yuejie Chi, Qing Zhao
TL;DR
This work analyzes distributed stochastic convex optimization under differential privacy, yielding a complete three-way trade-off among accuracy, communication, and privacy. It introduces Charter, a plane-cutting based algorithm that uses Vaidya's method with privacy-preserving gradient estimation and quantized communication, achieving an excess risk of $\widetilde{\mathcal{O}}\left( \dfrac{R\sigma_g}{\sqrt{MN}} + \dfrac{(R(1+\sigma_g)+\sigma_f)\sqrt{d}}{N\varepsilon_{\mathsf{DP}}\sqrt{M}} \right)$ and a communication cost of $\widetilde{\mathcal{O}}(d^2)$, thus matching a novel information-theoretic lower bound up to log factors. The lower bound shows that, for general convex objectives, any algorithm must incur at least $\Omega(d^2)$ bits of communication per client when aiming for centralized-optimal accuracy, highlighting an inherent efficiency bottleneck. Charter’s two-stage design—learning via gradient-collection with privacy safeguards and a verification stage for loss evaluation—enables order-optimal performance even with heterogeneous client data and without assuming identical data distributions. The results illuminate the fundamental frontier of distributed DP-SCO, suggesting new directions to reduce computational overhead and to extend to broader convex settings and privacy regimes. Overall, the paper provides a principled, tight characterization of the accuracy-communication-privacy landscape and a concrete method achieving it in distributed, differentially private stochastic optimization.
Abstract
We consider the problem of differentially private stochastic convex optimization (DP-SCO) in a distributed setting with $M$ clients, where each of them has a local dataset of $N$ i.i.d. data samples from an underlying data distribution. The objective is to design an algorithm to minimize a convex population loss using a collaborative effort across $M$ clients, while ensuring the privacy of the local datasets. In this work, we investigate the accuracy-communication-privacy trade-off for this problem. We establish matching converse and achievability results using a novel lower bound and a new algorithm for distributed DP-SCO based on Vaidya's plane cutting method. Thus, our results provide a complete characterization of the accuracy-communication-privacy trade-off for DP-SCO in the distributed setting.