FEDLAD: Federated Evaluation of Deep Leakage Attacks and Defenses

TL;DR

This work introduces FEDLAD, a unified benchmark for evaluating Deep Leakage gradient inversion attacks and defenses within Federated Learning. By enabling single- and multi-observation attacks, integrating multiple state-of-the-art methods under a common framework, and introducing the Recovery Consistency Index (RCI) alongside LPIPS-based recovery metrics, FEDLAD provides a realistic, reproducible platform for assessing privacy risks across datasets and training states. The study finds that GradInversion often yields the strongest recoveries in certain training regimes, while DLG and Inverting Gradients are less effective, and that computationally heavy approaches like Multiple Updates may be impractical on larger datasets. Defenses such as Gaussian DP, PRECODE, and DCS exhibit clear privacy-accuracy trade-offs, with PRECODE offering favorable balance between privacy and computation. Overall, FEDLAD aims to foster reproducibility and guide the development of more robust defenses in decentralized learning systems.

Abstract

Federated Learning is a privacy preserving decentralized machine learning paradigm designed to collaboratively train models across multiple clients by exchanging gradients to the server and keeping private data local. Nevertheless, recent research has revealed that the security of Federated Learning is compromised, as private ground truth data can be recovered through a gradient inversion technique known as Deep Leakage. While these attacks are crafted with a focus on applications in Federated Learning, they generally are not evaluated in realistic scenarios. This paper introduces the FEDLAD Framework (Federated Evaluation of Deep Leakage Attacks and Defenses), a comprehensive benchmark for evaluating Deep Leakage attacks and defenses within a realistic Federated context. By implementing a unified benchmark that encompasses multiple state-of-the-art Deep Leakage techniques and various defense strategies, our framework facilitates the evaluation and comparison of the efficacy of these methods across different datasets and training states. This work highlights a crucial trade-off between privacy and model accuracy in Federated Learning and aims to advance the understanding of security challenges in decentralized machine learning systems, stimulate future research, and enhance reproducibility in evaluating Deep Leakage attacks and defenses.

Figures (5)

• Figure 1: Overview of the interconnected training, attack, and evaluation processes that mutually influence each other during the training phase.
• Figure 2: SSIM-based reconstruction quality of attack on a CIFAR10 batch of 8 for 4 state-of-the-art attacks across 10,000 training iterations.
• Figure 3: Visual comparison for ImageNet recovery (batch size of 8) with state-of-the-art methods. From top to bottom; Ground Truth, DLG, IG, GradInversion and Multiple Updates.
• Figure 4: SSIM-based reconstruction quality against execution time with bubble size representing GPU VRAM for 4 state-of-the-art attacks with a batch of 8 CIFAR10 Images.
• Figure 5: SSIM Comparison for 5 state-of-the-art attacks with different CIFAR10 batch sizes.