Rényi Differential Privacy for Heavy-Tailed SDEs via Fractional Poincaré Inequalities
Benjamin Dupuis, Mert Gürbüzbalaban, Umut Şimşekli, Jian Wang, Sinan Yildirim, Lingjiong Zhu
TL;DR
This work presents the first Rényi differential privacy guarantees for learning dynamics driven by heavy-tailed, α-stable noise. By developing Rényi-flow analyses along Lévy-driven SDEs and replacing logarithmic Sobolev inequalities with fractional Poincaré inequalities, the authors obtain time-uniform or semi-uniform RDP bounds that exhibit weaker dimension dependence than prior results. The results cover both multifractal noise (Gaussian plus α-stable) and pure-jump α-stable cases, with explicit rates in terms of the noise scales, gradient-sensitivity, and sample size, and they extend to discrete-time heavy-tailed SGD. The analysis also provides stability results for fractional Poincaré inequalities under convolution and bi-Lipschitz mappings, offering theoretical guarantees for verifying the key assumptions in practice. Overall, the paper advances private learning with heavy-tailed dynamics and clarifies the tradeoffs between tail-heaviness, dimensionality, and privacy guarantees in realistic SGD-like algorithms.
Abstract
Characterizing the differential privacy (DP) of learning algorithms has become a major challenge in recent years. In parallel, many studies suggested investigating the behavior of stochastic gradient descent (SGD) with heavy-tailed noise, both as a model for modern deep learning models and to improve their performance. However, most DP bounds focus on light-tailed noise, where satisfactory guarantees have been obtained but the proposed techniques do not directly extend to the heavy-tailed setting. Recently, the first DP guarantees for heavy-tailed SGD were obtained. These results provide $(0,δ)$-DP guarantees without requiring gradient clipping. Despite casting new light on the link between DP and heavy-tailed algorithms, these results have a strong dependence on the number of parameters and cannot be extended to other DP notions like the well-established Rényi differential privacy (RDP). In this work, we propose to address these limitations by deriving the first RDP guarantees for heavy-tailed SDEs, as well as their discretized counterparts. Our framework is based on new Rényi flow computations and the use of well-established fractional Poincaré inequalities. Under the assumption that such inequalities are satisfied, we obtain DP guarantees that have a much weaker dependence on the dimension compared to prior art.