Towards hyperparameter-free optimization with differential privacy
Zhiqi Bu, Ruixuan Liu
TL;DR
This work tackles the challenge of hyperparameter tuning in differential privacy by introducing HyFreeDP, a hyperparameter-free DP training framework that privately and automatically updates the learning rate. It combines a privatized GeN-based learning-rate estimator with loss privatization to minimize clipping bias and employs end-to-end privacy accounting to determine noise levels, achieving DP guarantees with minimal overhead. The approach is validated across vision and language tasks, showing DP performance close to non-DP grid searches and superior stability relative to DP-tuning baselines. The practical impact is a scalable, end-to-end DP training method that reduces tuning effort and privacy risk while preserving strong performance and efficiency.
Abstract
Differential privacy (DP) is a privacy-preserving paradigm that protects the training data when training deep learning models. Critically, the performance of models is determined by the training hyperparameters, especially those of the learning rate schedule, thus requiring fine-grained hyperparameter tuning on the data. In practice, it is common to tune the learning rate hyperparameters through the grid search that (1) is computationally expensive as multiple runs are needed, and (2) increases the risk of data leakage as the selection of hyperparameters is data-dependent. In this work, we adapt the automatic learning rate schedule to DP optimization for any models and optimizers, so as to significantly mitigate or even eliminate the cost of hyperparameter tuning when applied together with automatic per-sample gradient clipping. Our hyperparameter-free DP optimization is almost as computationally efficient as the standard non-DP optimization, and achieves state-of-the-art DP performance on various language and vision tasks.