Deep Learning with Data Privacy via Residual Perturbation
Wenqi Tao, Huaming Ling, Zuoqiang Shi, Bao Wang
TL;DR
The paper tackles data privacy in deep learning by introducing residual perturbation, a Gaussian noise injection scheme applied at every residual mapping in ResNets and grounded in stochastic differential equation theory. By analyzing two SDE‑based strategies, the authors prove differential privacy guarantees and show a reduction in the generalization gap, while also achieving competitive or superior utility compared with DPSGD and enabling efficient training. They demonstrate through extensive experiments on IDC, MNIST, CIFAR10, and CIFAR100 that residual perturbation improves membership privacy (attacks approach random guessing) and can boost accuracy via model ensembles, with skip connections playing a crucial role. The work provides both theoretical DP/RDP results and practical insights into privacy‑utility tradeoffs, highlighting residual perturbation as a feasible path to private, accurate deep learning, albeit with opportunities for tighter DP bounds in future work.
Abstract
Protecting data privacy in deep learning (DL) is of crucial importance. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many existing mechanisms achieve privacy at the cost of significant utility degradation and computational overhead. In this paper, we propose a stochastic differential equation-based residual perturbation for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap of DL. Empirically, we show that residual perturbation is computationally efficient and outperforms the state-of-the-art differentially private stochastic gradient descent (DPSGD) in utility maintenance without sacrificing membership privacy.