SoK: Benchmarking Poisoning Attacks and Defenses in Federated Learning

TL;DR

This work tackles the vulnerability of federated learning to both data and model poisoning attacks, a problem complicated by fragmented and isolated evaluations. It introduces FLPoison, the first large-scale, unified benchmark that systematically compares 15 poisoning attacks and 17 defense strategies across common FL algorithms and data heterogeneity. The study provides a detailed taxonomy distinguishing DPAs, MPAs, and hybrids, and demonstrates how attack effectiveness and defense robustness vary with FL backbones (FedSGD vs. FedOpt) and data distributions (IID vs. non-IID). Key findings show that MPAs tend to be more potent under FedSGD while DPAs, especially targeted ones, are stronger under FedOpt, with certain defenses (Median, FLTrust, DnC, FLAME) offering robust protection across settings but still leaving gaps under non-IID data. The open-source FLPoison framework and its insights offer practical guidance for designing resilient FL systems and set a foundation for future cross-cutting defense research.

Abstract

Federated learning (FL) enables collaborative model training while preserving data privacy, but its decentralized nature exposes it to client-side data poisoning attacks (DPAs) and model poisoning attacks (MPAs) that degrade global model performance. While numerous proposed defenses claim substantial effectiveness, their evaluation is typically done in isolation with limited attack strategies, raising concerns about their validity. Additionally, existing studies overlook the mutual effectiveness of defenses against both DPAs and MPAs, causing fragmentation in this field. This paper aims to provide a unified benchmark and analysis of defenses against DPAs and MPAs, clarifying the distinction between these two similar but slightly distinct domains. We present a systematic taxonomy of poisoning attacks and defense strategies, outlining their design, strengths, and limitations. Then, a unified comparative evaluation across FL algorithms and data heterogeneity is conducted to validate their individual and mutual effectiveness and derive key insights for design principles and future research. Along with the analysis, we frame our work to a unified benchmark, FLPoison, with high modularity and scalability to evaluate 15 representative poisoning attacks and 17 defense strategies, facilitating future research in this domain. Code is available at https://github.com/vio1etus/FLPoison.

Figures (10)

• Figure 1: Taxonomies of Poisoning Attacks in Federated Learning
• Figure 2: Overview of Our FLPoison
• Figure 3: Illustration of Poisoning Attack Principles
• Figure 4: Visualization of statistical heterogeneity across 20 clients for our IID and Non-IID partitions on CIFAR-10 dataset, where the x-axis indicates client IDs, the y-axis shows the number of training samples on that client, and colors represent label classes.
• Figure 5: Comparison of Attacks in terms of FL Algorithms