iDLG: Improved Deep Leakage from Gradients

TL;DR

The paper shows that ground-truth labels can be deterministically inferred from gradients in models trained with cross-entropy over one-hot labels, enabling a simplified and reliable data-extraction method. It introduces Improved DLG (iDLG), which first extracts the true label from shared gradients and then reconstructs data by gradient matching with a simpler objective. Empirically, iDLG achieves 100% label extraction accuracy and higher data fidelity across MNIST, CIFAR-100, and LFW compared to the original DLG, with faster convergence especially on challenging datasets. This work underscores a significant privacy risk in gradient sharing and motivates the development of defenses against gradient-based leakage in distributed learning systems.

Abstract

It is widely believed that sharing gradients will not leak private training data in distributed learning systems such as Collaborative Learning and Federated Learning, etc. Recently, Zhu et al. presented an approach which shows the possibility to obtain private training data from the publicly shared gradients. In their Deep Leakage from Gradient (DLG) method, they synthesize the dummy data and corresponding labels with the supervision of shared gradients. However, DLG has difficulty in convergence and discovering the ground-truth labels consistently. In this paper, we find that sharing gradients definitely leaks the ground-truth labels. We propose a simple but reliable approach to extract accurate data from the gradients. Particularly, our approach can certainly extract the ground-truth labels as opposed to DLG, hence we name it Improved DLG (iDLG). Our approach is valid for any differentiable model trained with cross-entropy loss over one-hot labels. We mathematically illustrate how our method can extract ground-truth labels from the gradients and empirically demonstrate the advantages over DLG.

Figures (2)

• Figure 1: The Fidelity comparison of DLG zhu19deep and iDLG on three datasets. The x-axis denotes the (MSE) threshold of good fidelity. From left to right, the threshold decreases and the fidelity requirement improves. Obviously, the proposed iDLG consistently outperforms DLG in recovering data with significant margin on three tasks. The advantage of iDLG is remarkable on the hard task of LFW.
• Figure 2: Example of the training process of DLG (left) and iDLG (right) on LFW face dataset. The first image is the (original) private training image, and the followings are the extracted images at different training iterations. It is clear that the training of iDLG is easier to converge.