On Optimal Hyperparameters for Differentially Private Deep Transfer Learning

TL;DR

This paper addresses how to optimally set two DP hyperparameters, the clipping bound $C$ and batch size $B$, during differentially private transfer learning. It challenges the common practice of fixing $(C,B)$ across tasks by showing that optimal values shift with privacy level, backbone capability, and compute budgets due to changes in gradient distributions and DP noise accumulation. A key theoretical contribution is the exact optimal clipping bound $C^*$ that minimizes mean squared error: $C^* = \frac{N_{C^*}^T G_{C^*}}{N_{C^*}^T N_{C^*} + \sigma^2 d}$, paired with insights that clipping acts as gradient re-weighting and that cumulative DP noise governs batch-size effectiveness under fixed epochs. The authors propose practical guidance: tune $C$ and $B$ jointly to match problem difficulty and privacy constraints, leveraging a minimum-step plus cumulative-noise criterion and recognizing the plateau behavior of DP noise in fixed-epoch regimes. These findings have practical implications for DP deployments, suggesting that adaptive hyperparameter strategies can significantly improve DP transfer learning performance, especially on harder problems or constrained compute budgets.

Abstract

Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the-art approach for training large models under privacy constraints. We focus on two key hyperparameters in this setting: the clipping bound $C$ and batch size $B$. We show a clear mismatch between the current theoretical understanding of how to choose an optimal $C$ (stronger privacy requires smaller $C$) and empirical outcomes (larger $C$ performs better under strong privacy), caused by changes in the gradient distributions. Assuming a limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning $B$ do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. We also highlight how the common practice of using a single $(C,B)$ setting across tasks can lead to suboptimal performance. We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as a form of gradient re-weighting and examining cumulative DP noise.

Figures (22)

• Figure 1: Macro accuracy heatmaps under increasing learning-problem difficulty (left to right): strong model with loose privacy, weaker model with same privacy, and weaker model with tight privacy. Experiments use DP-Adam with $\delta=10^{-5}$ on SUN397 ($\text{'Full'}{=}87002$), averaged over 5 seeds with 8 epochs each. Learning rates are tuned separately for each configuration using a fixed grid. No single clipping bound or batch size performs best across all settings.
• Figure 2: Accuracy difference to the best, mean over 5 repeats with min--max bands; SUN397 dataset, 8 epochs, $\delta{=}10^{-5}$. Left: optimal clipping constant is larger with tighter privacy (small $\varepsilon$). Right: low-capability pretrained backbone (ViT-Tiny) performs better with larger clipping bounds compared to a high-capability model (ViT-Base). In both cases, harder tasks (orange) shift the optimal clipping bound upward, while easier tasks (blue) tolerate or even benefit from smaller bounds.
• Figure 3: Gradient norm distributions after training, ViT-Tiny on SUN397, 8 epochs; $\delta = 10^{-5}$. Wider sections of each violin plot indicate higher density; black lines denote medians. As $\varepsilon$ decreases (left to right in each subplot), the distributions shift significantly upward.
• Figure 4: Per-class effects of gradient clipping under increasing task difficulty (left to right), SUN397, 8 epochs; $\delta{=}10^{-5}$. Top: relative retained weight after clipping, computed per class and normalized to the class with the largest retained weight in the baseline ($C{=}1$). Flatter and higher curves indicate that gradients are preserved more uniformly across classes, while steep drops reveal that hard classes suffer disproportionately from clipping. Bottom: change in per-class accuracy relative to the baseline (higher is better). When difficulty increases, smaller clipping bounds ($C{=}0.1$) slightly help easy classes but sharply degrade hard classes, while larger bounds ($C{=}42$) mildly reduce easy-class accuracy but substantially improve the hardest classes. Small clipping bounds risk over-clipping gradients from hard classes, while large bounds better preserve gradient signal across the classes under high learning-problem difficulty.
• Figure 5: Comparison of AUTO-SbuAutomaticClippingDifferentially2023 and properly tuned flat clipping, mean accuracy with min--max bands over 3 repeats, ViT-Tiny model, 8 epochs training for SUN397 and CIFAR-100, 32 epochs for Cassava; $\delta = 10^{-5}$. Batch size and learning rate were tuned over a fixed grid for both methods. AUTO-S performs notably worse on harder datasets (SUN397, Cassava), especially under tight privacy, as predicted by our analysis. See \ref{['app:autoclip-vs-tuned']} for exact values.

Theorems & Definitions (5)

• Definition 2.1
• Theorem 5.1
• proof
• Theorem D.1
• proof