Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization

Abstract

Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning, allowing multiple clients to collaboratively train a global model by transmitting locally computed gradients to a central server without exposing their private data. Nonetheless, recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage, e.g., an attacker can invert shared gradients to reconstruct sensitive data by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. However, existing attacks simply perform gradient inversion in the latent space of the GAN model, which limits their expression ability and generalizability. To tackle these challenges, we propose \textbf{G}radient \textbf{I}nversion over \textbf{F}eature \textbf{D}omains (GIFD), which disassembles the GAN model and searches the hierarchical features of the intermediate layers. Instead of optimizing only over the initial latent code, we progressively change the optimized layer, from the initial latent space to intermediate layers closer to the output images. In addition, we design a regularizer to avoid unreal image generation by adding a small ${l_1}$ ball constraint to the searching range. We also extend GIFD to the out-of-distribution (OOD) setting, which weakens the assumption that the training sets of GANs and FL tasks obey the same data distribution. Furthermore, we consider the challenging OOD scenario of label inconsistency and propose a label mapping technique as an effective solution. Extensive experiments demonstrate that our method can achieve pixel-level reconstruction and outperform competitive baselines across a variety of FL scenarios.

Figures (10)

• Figure 1: The reconstructed results of our proposed GIFD on ImageNetdeng2009imagenet and FFHQkarras2019style. The first column contains the randomly initialized images generated by GAN models. The next two columns show the reconstruction samples of the latent space search and our proposed GIFD.
• Figure 2: Overview of our proposed GIFD attack. The intermediate layer optimizer minimizes the matching loss computed from the dummy gradients and the shared gradients from the victim client to update the latent vector and the intermediate features successively. The image fidelity regularization helps improve the quality of generated images. The reconstructed image from the layer with the corresponding least gradient matching loss is selected as the final output.
• Figure 3: An illustration of our proposed label mapping mechanism. We first conduct only a small number of iterations based on the initially inferred label for a coarse reconstruction $\mathbf{\hat{x}}$. It is then fed into $f_m(\cdot)$ for a refined label, which serves as the conditional input for GAN models in subsequent fine-grained attack. This technique corrects the label inconsistency and offers a more aligned guidance in the semantic space, further enhancing the attack effectiveness.
• Figure 4: Comparison of PSNR mean on BigGAN and StyleGAN2 under different values of hyper-parameter $K$ (i.e., the last intermediate layer to optimize). Notably, the figures exclude the results where the corresponding values are below the starting point of the y-axis.
• Figure 5: Qualitative results of different methods on ImageNet and FFHQ.