Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners
Rachel Redberg, Antti Koskela, Yu-Xiang Wang
TL;DR
This paper addresses the privacy-utility trade-off in differentially private learning for convex generalized linear models by reviving objective perturbation with two tight privacy analyses: a privacy-profiles-based $(\epsilon, \delta)$-DP bound and a Rényi differential privacy bound. It also extends Approximate Minima Perturbation to unbounded gradient losses via gradient clipping, links the approach to SVRG for fast optimization, and establishes a subquadratic $O(n \log n)$ computational guarantee. The authors show that, when accounting for the privacy cost of hyperparameter tuning, objective perturbation can be competitive with DP-SGD on GLMs, supported by empirical results on standard datasets. Overall, the work broadens the practical applicability of objective perturbation and provides tighter, modern privacy accounting tools for private learning.
Abstract
In the arena of privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) has outstripped the objective perturbation mechanism in popularity and interest. Though unrivaled in versatility, DP-SGD requires a non-trivial privacy overhead (for privately tuning the model's hyperparameters) and a computational complexity which might be extravagant for simple models such as linear and logistic regression. This paper revamps the objective perturbation mechanism with tighter privacy analyses and new computational tools that boost it to perform competitively with DP-SGD on unconstrained convex generalized linear problems.