Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature

TL;DR

This work investigates input loss curvature, defined as $Curv_\phi(z_i,S)=\mathrm{tr}(\nabla^2_{z_i} \ell(h^{\phi}_S,z_i))$, as a signal for distinguishing train from test samples and for membership inference in deep networks. It develops a theoretical framework that bounds train-test distinguishability via KL divergence, linking bounds to the privacy parameter $\epsilon$ and training size $m$, and shows curvature-based MIA can surpass probability-based methods on sufficiently large datasets. To enable privacy testing in black-box settings, the authors introduce a zero-order curvature estimator that uses shadow models and likelihood-ratio tests (LR/NLL) to perform MIA without access to model parameters. Empirical validation on CIFAR10/100 and ImageNet demonstrates superior performance of curvature-based MIAs over state-of-the-art techniques, and confirms the predicted dependence on dataset size, privacy, and dataset composition. These findings advance understanding of how input geometry relates to memorization and privacy and provide practical tools for evaluating privacy-preserving techniques in vision models.

Abstract

In this paper, we explore the properties of loss curvature with respect to input data in deep neural networks. Curvature of loss with respect to input (termed input loss curvature) is the trace of the Hessian of the loss with respect to the input. We investigate how input loss curvature varies between train and test sets, and its implications for train-test distinguishability. We develop a theoretical framework that derives an upper bound on the train-test distinguishability based on privacy and the size of the training set. This novel insight fuels the development of a new black box membership inference attack utilizing input loss curvature. We validate our theoretical findings through experiments in computer vision classification tasks, demonstrating that input loss curvature surpasses existing methods in membership inference effectiveness. Our analysis highlights how the performance of membership inference attack (MIA) methods varies with the size of the training set, showing that curvature-based MIA outperforms other methods on sufficiently large datasets. This condition is often met by real datasets, as demonstrated by our results on CIFAR10, CIFAR100, and ImageNet. These findings not only advance our understanding of deep neural network behavior but also improve the ability to test privacy-preserving techniques in machine learning.

Figures (7)

• Figure 1: Visualizing low and high input curvature samples from a ResNet50 trained on ImageNet. Low input curvature training set images are prototypical and have lots of support in the trainset, while high input curvature train set examples have less support and are atypical. Test set examples lie around the training set images in higher curvature regions.
• Figure 2: Marginal curvature score histogram for all images in ImageNet when samples in train set vs test set.
• Figure 3: Conditional curvature scores for a single image from ImageNet when using 56 shadow models.
• Figure 4: Comparing our method against existing techniques at low FPR. The proposed parametric Curv LR technique has the highest TPR at very low FPR.
• Figure 5: Validating the upper bound from Theorem \ref{['th:pc_dkl']} by fitting the MIA performance (AUROC of Curv LR) on DP-SGD trained models for various privacy parameters $\epsilon$ values.

Theorems & Definitions (5)

• Theorem 4.1: Privacy bounds Train-Test KL Divergence
• Theorem 4.2: Dataset Size and Privacy bound Curvature KL Divergence
• Theorem 4.3: Dataset Size and Curvature MIA Performance
• Lemma A.1: Upper bound on Input Loss Curvature
• Lemma A.2