FedSecurity: Benchmarking Attacks and Defenses in Federated Learning and Federated LLMs

TL;DR

FedSecurity provides an end-to-end benchmark for adversarial attacks and defenses in FL and federated LLMs, addressing the lack of unified evaluation frameworks. It introduces FedAttacker and FedDefender to simulate a broad spectrum of attacks (model poisoning, data poisoning, data reconstruction) and defenses (before/on/after-aggregation), with flexible YAML configurations and API-based extensions. The benchmark demonstrates the trade-offs between defense effectiveness and potential accuracy loss, and highlights the robustness of approaches like m-Krum across FL and LLM contexts, including real-world edge deployments. By supporting diverse models, optimizers, and real-world scenarios, FedSecurity offers a practical platform to benchmark, compare, and advance secure federated learning systems.

Abstract

This paper introduces FedSecurity, an end-to-end benchmark that serves as a supplementary component of the FedML library for simulating adversarial attacks and corresponding defense mechanisms in Federated Learning (FL). FedSecurity eliminates the need for implementing the fundamental FL procedures, e.g., FL training and data loading, from scratch, thus enables users to focus on developing their own attack and defense strategies. It contains two key components, including FedAttacker that conducts a variety of attacks during FL training, and FedDefender that implements defensive mechanisms to counteract these attacks. FedSecurity has the following features: i) It offers extensive customization options to accommodate a broad range of machine learning models (e.g., Logistic Regression, ResNet, and GAN) and FL optimizers (e.g., FedAVG, FedOPT, and FedNOVA); ii) it enables exploring the effectiveness of attacks and defenses across different datasets and models; and iii) it supports flexible configuration and customization through a configuration file and some APIs. We further demonstrate FedSecurity's utility and adaptability through federated training of Large Language Models (LLMs) to showcase its potential on a wide range of complex applications.

Figures (17)

• Figure 1: Examples of attack and defense configurations.
• Figure 2: FedSecurity overview. FedSecurity enables injecting attacks/defenses (shown in red/green) at various stages of FL at the clients and at the server.
• Figure 3: Attack comparison.
• Figure 4: Defense comparison.
• Figure 5: Label flipping exps.