Technical Report: Full Version of Analyzing and Optimizing Perturbation of DP-SGD Geometrically

TL;DR

This work addresses the privacy-utility trade-off in training with DP-SGD by revealing that DP noise biased the gradient direction, which standard SGD optimizations cannot effectively counteract. It introduces GeoDP, a geometric perturbation method that operates in hyper-spherical coordinates to perturb gradient direction and magnitude separately, achieving unbiased directional noise within a reduced directional subspace controlled by a bounding factor $\beta$. Theoretical analysis shows DP-SGD perturbations are sub-optimal in a geometric sense, while GeoDP improves the descent trajectory under DP guarantees; empirical results on MNIST, CIFAR-10, and a synthetic gradient dataset across LR, CNN, and ResNet tasks demonstrate improved directional preservation and competitive performance, with runtime trade-offs that can be mitigated by hardware or parallelization. Overall, GeoDP offers a general, DP-compliant approach to enhancing the efficiency of private SGD and suggests avenues for extension to other optimizers and federated learning.

Abstract

Differential privacy (DP) has become a prevalent privacy model in a wide range of machine learning tasks, especially after the debut of DP-SGD. However, DP-SGD, which directly perturbs gradients in the training iterations, fails to mitigate the negative impacts of noise on gradient direction. As a result, DP-SGD is often inefficient. Although various solutions (e.g., clipping to reduce the sensitivity of gradients and amplifying privacy bounds to save privacy budgets) are proposed to trade privacy for model efficiency, the root cause of its inefficiency is yet unveiled. In this work, we first generalize DP-SGD and theoretically derive the impact of DP noise on the training process. Our analysis reveals that, in terms of a perturbed gradient, only the noise on direction has eminent impact on the model efficiency while that on magnitude can be mitigated by optimization techniques, i.e., fine-tuning gradient clipping and learning rate. Besides, we confirm that traditional DP introduces biased noise on the direction when adding unbiased noise to the gradient itself. Overall, the perturbation of DP-SGD is actually sub-optimal from a geometric perspective. Motivated by this, we design a geometric perturbation strategy GeoDP within the DP framework, which perturbs the direction and the magnitude of a gradient, respectively. By directly reducing the noise on the direction, GeoDP mitigates the negative impact of DP noise on model efficiency with the same DP guarantee. Extensive experiments on two public datasets (i.e., MNIST and CIFAR-10), one synthetic dataset and three prevalent models (i.e., Logistic Regression, CNN and ResNet) confirm the effectiveness and generality of our strategy.

Figures (6)

• Figure 1: Comparing MSEs of GeoDP and DP on preserving directions and values of gradients under synthetic dataset (composed of gradients from CNN training, as introduced in Section \ref{['subsec:setup']}). While $\theta$ and $g$ label the MSE of perturbed directions and gradients themselves, experimental results confirm that GeoDP achieves smaller MSEs on perturbed directions (i.e., the red line is below the black one), while sacrificing the accuracy of perturbed gradients (i.e., the green line is above the blue one). In general, GeoDP better preserves directions of gradients while traditional DP only excels in preserving numerical values of gradients.
• Figure 2: Coordinates Conversions in Three-dimensional Space
• Figure 3: GeoDP vs. DP on Preserving Gradients under Various Parameters on Synthetic Dataset
• Figure 4: The Effectiveness of Bounding Factor
• Figure 5: GeoDP versus DP on Logistic Regression under MNIST dataset

Theorems & Definitions (9)

• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof