An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses
Hao Liang, Wanrong Zhang, Xinlei He, Kaishun Wu, Hong Xing
TL;DR
The paper tackles the challenge of quantifying privacy loss in differentially private SGD without heavily restrictive assumptions. It introduces a Noisy Smooth-Reduction framework and shift Rényi divergence to derive closed-form $(\alpha,\varepsilon)$-RDP bounds for DPSGD-GC and DPSGD-DC under $L$-smooth, non-convex losses, in both unbounded and bounded domains. The authors also provide accompanying utility analyses and show that a smaller bounded domain diameter $D$ improves both privacy and utility under certain conditions, with concrete Big-O bounds and mu-strongly convex case results. Empirical validation via membership inference attacks confirms the theoretical insights and demonstrates practical privacy-utility trade-offs across batch sizes and domain diameters. Overall, the work advances rigorous, convergent privacy analysis for private optimization and guides design choices in privacy-utility trade-offs for DPSGD variants.
Abstract
Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to protect sensitive data during the training of machine learning models, but its privacy guarantee often comes at a large cost of model performance due to the lack of tight theoretical bounds quantifying privacy loss. While recent efforts have achieved more accurate privacy guarantees, they still impose some assumptions prohibited from practical applications, such as convexity and complex parameter requirements, and rarely investigate in-depth the impact of privacy mechanisms on the model's utility. In this paper, we provide a rigorous privacy characterization for DPSGD with general L-smooth and non-convex loss functions, revealing converged privacy loss with iteration in bounded-domain cases. Specifically, we track the privacy loss over multiple iterations, leveraging the noisy smooth-reduction property, and further establish comprehensive convergence analysis in different scenarios. In particular, we show that for DPSGD with a bounded domain, (i) the privacy loss can still converge without the convexity assumption, (ii) a smaller bounded diameter can improve both privacy and utility simultaneously under certain conditions, and (iii) the attainable big-O order of the privacy utility trade-off for DPSGD with gradient clipping (DPSGD-GC) and for DPSGD-GC with bounded domain (DPSGD-DC) and mu-strongly convex population risk function, respectively. Experiments via membership inference attack (MIA) in a practical setting validate insights gained from the theoretical results.