Towards Eliminating Hard Label Constraints in Gradient Inversion Attacks

TL;DR

This paper tackles gradient inversion attacks under realistic soft-label training where label smoothing and mixup obscure one-hot labels. It introduces an analytical method that recovers augmented labels and the last-layer input features from a single input gradient by solving for a scalar multiplier $\lambda_r$ so that the pseudo-label $\hat{y}$ aligns with a variance-based loss $\mathcal{L}_{label}$, and proves a normalization constraint via Theorem 1 that $\sum_i \hat{y}_i = 1$. Empirically, the method yields label-recovery accuracy above 95% across CIFAR-100, Flowers-17, and ImageNet, and enhances FCN and CNN image reconstruction compared to prior attacks. The results imply that soft-label regimes do not fully prevent gradient inversion in federated learning, highlighting privacy implications and motivating defenses.

Abstract

Gradient inversion attacks aim to reconstruct local training data from intermediate gradients exposed in the federated learning framework. Despite successful attacks, all previous methods, starting from reconstructing a single data point and then relaxing the single-image limit to batch level, are only tested under hard label constraints. Even for single-image reconstruction, we still lack an analysis-based algorithm to recover augmented soft labels. In this work, we change the focus from enlarging batchsize to investigating the hard label constraints, considering a more realistic circumstance where label smoothing and mixup techniques are used in the training process. In particular, we are the first to initiate a novel algorithm to simultaneously recover the ground-truth augmented label and the input feature of the last fully-connected layer from single-input gradients, and provide a necessary condition for any analytical-based label recovery methods. Extensive experiments testify to the label recovery accuracy, as well as the benefits to the following image reconstruction. We believe soft labels in classification tasks are worth further attention in gradient inversion attacks.

Figures (7)

• Figure 1: Variance loss under untrained and pretrained ResNet18. The gradient-based optimizer may find local minima at approximately 2.2372 while the ground-truth is 4.2745.
• Figure 2: Recovered label distribution and variance loss given same gradient information. When we alter the scalar $\lambda_r$, the class probability would vary. If we have no more information about label distribution, all labels except the left-top and right-bottom ones could be the right labels generating exactly the same gradients.
• Figure 3: Image reconstruction comparisons on FCN-4 network. Images with label smoothing are compressed to $64\times64$ due to CUDA memory limitation in optimization-based methods. For more samples with mixup augmentation please refer to Appendix \ref{['reco']}.
• Figure 4: Recovered images with label smoothing augmentations from FCN-4. All 100 images are randomly picked with unknown smoothing probability.
• Figure 5: Recovered images with mixup augmentations from FCN-4. All 100 mixed images are randomly picked with unknown mixup probability.