SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training

TL;DR

This work addresses fairness in differentially private training by revising the clipping mechanism in DP-SGD. It introduces SoftAdaClip, which uses a smooth $\tanh$-based gradient transformation combined with adaptive clipping to bound sensitivity while preserving subgroup learning signals. Across MIMIC-III, eICU, and Adult Income datasets, SoftAdaClip reduces subgroup disparities substantially, with statistically significant improvements over DP-SGD and Adaptive-DPSGD, and maintains strong $$(\epsilon,\delta)$$ guarantees. The results suggest that integrating smooth transformations with adaptive mechanisms is a promising path toward fair and private model training, though performance gains can be dataset-dependent and require careful hyperparameter tuning.

Abstract

Differential privacy (DP) provides strong protection for sensitive data, but often reduces model performance and fairness, especially for underrepresented groups. One major reason is gradient clipping in DP-SGD, which can disproportionately suppress learning signals for minority subpopulations. Although adaptive clipping can enhance utility, it still relies on uniform hard clipping, which may restrict fairness. To address this, we introduce SoftAdaClip, a differentially private training method that replaces hard clipping with a smooth, tanh-based transformation to preserve relative gradient magnitudes while bounding sensitivity. We evaluate SoftAdaClip on various datasets, including MIMIC-III (clinical text), GOSSIS-eICU (structured healthcare), and Adult Income (tabular data). Our results show that SoftAdaClip reduces subgroup disparities by up to 87% compared to DP-SGD and up to 48% compared to Adaptive-DPSGD, and these reductions in subgroup disparities are statistically significant. These findings underscore the importance of integrating smooth transformations with adaptive mechanisms to achieve fair and private model training.

Figures (7)

• Figure 1: Comparison of hard clipping ($\min(1, x)$) and SoftAdaClip ($\tanh(x)$). Hard clipping maps all gradients above the threshold to the same value, while SoftAdaClip smoothly compresses them, preserving differences.
• Figure 2: Comparison of loss gaps between subgroups across different datasets and clipping strategies. The figure presents the subgroup loss disparities for three differentially private training methods: DPSGD, andrew2021differentially Adaptive-DPSGD, and SoftAdaClip.. Error bars represent ±1 standard error of the mean, computed across five random seeds.
• Figure 3: Comparison of train, validation, and test losses across datasets using DPSGD, andrew2021differentially Adaptive-DPSGD, and SoftAdaClip. Results are averaged over five random seeds. Error bars represent ±1 standard error of the mean.
• Figure 4: Comparison of loss gaps between subgroups in the Income Simple model using clipping thresholds of 0.01 and 0.05 across three differentially private training methods: DPSGD, andrew2021differentially Adaptive-DPSGD, and SoftAdaClip.
• Figure 5: Comparison of train, validation, and test losses in the Income Simple model using clipping thresholds of 0.01 and 0.05 across three differentially private training methods: DPSGD, andrew2021differentially Adaptive-DPSGD, and SoftAdaClip.