When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning

TL;DR

This work tackles the clipping bottleneck in differentially private deep learning by recasting clipping as a closed-loop control problem. It introduces WW-DP-SGD, which leverages a WeightWatcher-style spectral tail exponent $oldsymbol{}$ computed from a private weight matrix to gauge training health and adapt the clipping threshold $C_t$ via a log-domain, bounded controller; the method operates as post-processing of DP outputs and does not increase privacy loss under standard accounting. Empirical results across vision and tabular tasks show WW-DP-SGD consistently improves utility and stability over fixed clipping and top adaptive baselines, with modest runtime overhead and robustness to distribution shifts. The approach provides practical guidance on probe-layer selection, smoothing, and controller parameters, and it opens avenues for further theoretical linking between spectral properties and DP optimization dynamics.

Abstract

Privacy-preserving training on sensitive data commonly relies on differentially private stochastic optimization with gradient clipping and Gaussian noise. The clipping threshold is a critical control knob: if set too small, systematic over-clipping induces optimization bias; if too large, injected noise dominates updates and degrades accuracy. Existing adaptive clipping methods often depend on per-example gradient norm statistics, adding computational overhead and introducing sensitivity to datasets and architectures. We propose a control-driven clipping strategy that adapts the threshold using a lightweight, weight-only spectral diagnostic computed from model parameters. At periodic probe steps, the method analyzes a designated weight matrix via spectral decomposition and estimates a heavy-tailed spectral indicator associated with training stability. This indicator is smoothed over time and fed into a bounded feedback controller that updates the clipping threshold multiplicatively in the log domain. Because the controller uses only parameters produced during privacy-preserving training, the resulting threshold updates are post-processing and do not increase privacy loss beyond that of the underlying DP optimizer under standard composition accounting.

Figures (3)

• Figure 1: WW-DP-SGD: closed-loop differentially private optimization via spectral feedback. DP-SGD performs Poisson subsampling, per-example gradient clipping with threshold $C_t$, and Gaussian noise injection to produce a noisy update $\tilde{g}_t$ with formal $(\varepsilon,\delta)$ guarantees. A WeightWatcher-style spectral diagnostic is computed periodically from a fixed probe weight matrix $W(\theta_t)$, yielding a heavy-tailed exponent $\zeta_t$ that is smoothed and regulated toward a target spectral health zone. The resulting control signal adaptively updates the clipping threshold in log-space. Crucially, the feedback loop operates exclusively on model weights and released quantities, constituting post-processing and therefore preserving the original DP accounting.
• Figure 2: Ablation on probe-layer selection for the spectral proxy on CIFAR-10 (ResNet) under matched privacy. Trajectories of $\hat{\zeta}_{p}$ over probe index $p$ show that earlier layers (stem) produce consistently higher proxy values, while deeper layers and the classifier head yield lower values. The full-model median provides a balanced intermediate signal.
• Figure 3: Calibration (ECE) vs. privacy budget $\varepsilon$ on MNIST. Lower values indicate better calibration. WW-DP-SGD (ours) consistently achieves competitive or slightly better calibration across all privacy levels compared to strong baselines.

Theorems & Definitions (5)

• Definition 2.1: Differential Privacy (DP)
• Lemma 4.1: Adaptive clipping is safe
• proof : Proof sketch
• Theorem 4.2: DP guarantee for WW-DP-SGD
• proof : Proof sketch