Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bayesian Robust Aggregation for Federated Learning

Aleksandr Karakulev, Usama Zafar, Salman Toor, Prashant Singh

TL;DR

This work tackles Byzantine robustness in federated learning by introducing a Bayesian robust aggregation that marginalizes over client honesty probabilities. It casts aggregation as maximizing a marginal likelihood via an ELBO with a variational posterior over binary indicators, yielding EM-style updates for the global weight, scale, and client posteriors. The approach adapts to the number of malicious clients without extra hyperparameters and achieves strong performance against both untargeted and backdoor attacks across MNIST, FMNIST, and CIFAR-10, often matching or surpassing specialized defenses like Krum. Importantly, it preserves normal performance in benign settings, enabling practical deployment in dynamic adversarial environments.

Abstract

Federated Learning enables collaborative training of machine learning models on decentralized data. This scheme, however, is vulnerable to adversarial attacks, when some of the clients submit corrupted model updates. In real-world scenarios, the total number of compromised clients is typically unknown, with the extent of attacks potentially varying over time. To address these challenges, we propose an adaptive approach for robust aggregation of model updates based on Bayesian inference. The mean update is defined by the maximum of the likelihood marginalized over probabilities of each client to be `honest'. As a result, the method shares the simplicity of the classical average estimators (e.g., sample mean or geometric median), being independent of the number of compromised clients. At the same time, it is as effective against attacks as methods specifically tailored to Federated Learning, such as Krum. We compare our approach with other aggregation schemes in federated setting on three benchmark image classification data sets. The proposed method consistently achieves state-of-the-art performance across various attack types with static and varying number of malicious clients.

Bayesian Robust Aggregation for Federated Learning

TL;DR

This work tackles Byzantine robustness in federated learning by introducing a Bayesian robust aggregation that marginalizes over client honesty probabilities. It casts aggregation as maximizing a marginal likelihood via an ELBO with a variational posterior over binary indicators, yielding EM-style updates for the global weight, scale, and client posteriors. The approach adapts to the number of malicious clients without extra hyperparameters and achieves strong performance against both untargeted and backdoor attacks across MNIST, FMNIST, and CIFAR-10, often matching or surpassing specialized defenses like Krum. Importantly, it preserves normal performance in benign settings, enabling practical deployment in dynamic adversarial environments.

Abstract

Federated Learning enables collaborative training of machine learning models on decentralized data. This scheme, however, is vulnerable to adversarial attacks, when some of the clients submit corrupted model updates. In real-world scenarios, the total number of compromised clients is typically unknown, with the extent of attacks potentially varying over time. To address these challenges, we propose an adaptive approach for robust aggregation of model updates based on Bayesian inference. The mean update is defined by the maximum of the likelihood marginalized over probabilities of each client to be `honest'. As a result, the method shares the simplicity of the classical average estimators (e.g., sample mean or geometric median), being independent of the number of compromised clients. At the same time, it is as effective against attacks as methods specifically tailored to Federated Learning, such as Krum. We compare our approach with other aggregation schemes in federated setting on three benchmark image classification data sets. The proposed method consistently achieves state-of-the-art performance across various attack types with static and varying number of malicious clients.
Paper Structure (24 sections, 1 theorem, 25 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 24 sections, 1 theorem, 25 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Experimental Settings
  4. Datasets and Model
  5. Federated Setup
  6. Attack Setup
  7. Compared Defense Baselines
  8. Evaluation Metrics
  9. Results
  10. Evaluating Configurations 1--3
  11. Ablation Study
  12. Limitations
  13. Conclusion
  14. Detailed Experiment Setup
  15. Datasets and Models
  16. ...and 9 more sections

Key Result

Proposition 2.1

For any vectors $\bm{w}_1, ..., \bm{w}_K \in \mathbb{R}^d$ and any set $B \subseteq \{1, ..., K\}$ of cardinality $K - M$, where $M < K/2$, denote $\overline{\bm{w}}_B = \sum_{k \in B} w_k/\vert B \vert$. Then for solution $\overline{\bm{w}}_S$ of eq:set-form, we have where $\kappa = 4 \left( 1 + \dfrac{M}{K - 2M} \right)$.

Figures (4)

  • Figure 1: Federated Learning for classifying FMNIST data using 25 rounds and 10 clients. Attacks are dynamic: in some communication rounds, malicious clients submit honest updates (upper heatmap). Our aggregation method estimates probability for each client of being 'benign' (lower heatmap), which results in a global model with high test accuracy (bottom).
  • Figure 2: Ablation study. (a) Test accuracy (ACC) under Sign Flip with fraction of malicious clients varying from 0 to 40% and (b) attack success rate (ASR) under Backdoor attack with fraction of malicious clients varying from 10% to 40%.
  • Figure 3: Examples of backdoor-triggered samples from (a) MNIST, (b) Fashion-MNIST, and (c) CIFAR-10 datasets.
  • Figure 4: Ablation study. Test accuracy (ACC) under (a) Random and (b) Label-flipping attacks, with fraction of malicious clients varying from 0 to 40%.

Theorems & Definitions (2)

  • Proposition 2.1
  • proof