Differential Privacy Regularization: Protecting Training Data Through Loss Function Regularization

TL;DR

The paper addresses privacy leakage from training data in neural networks and critiques gradient-noise based DP approaches like DP-SGD for potentially diminishing utility. It introduces PDP-SGD, a loss-regularization strategy where privacy is enforced through an input- and parameter-dependent regularization term rather than explicit gradient perturbation. The authors argue that the PDP regularization captures DP effects via a proportional input-dependent term that can be integrated with conventional L2 regularization, potentially improving the privacy–utility trade-off. They also highlight possible efficiency gains by avoiding explicit gradient noise and maintaining compatibility with standard SGD optimizers, thereby broadening differential privacy applications in large models and LLMs.

Abstract

Training machine learning models based on neural networks requires large datasets, which may contain sensitive information. The models, however, should not expose private information from these datasets. Differentially private SGD [DP-SGD] requires the modification of the standard stochastic gradient descent [SGD] algorithm for training new models. In this short paper, a novel regularization strategy is proposed to achieve the same goal in a more efficient manner.