Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks

TL;DR

This paper tackles backdoor threats in federated learning under non-IID data by introducing Robust Knowledge Distillation (RKD), a defense that combines cosine-similarity based clustering with HDBSCAN, median-based model selection, and knowledge distillation from a vetted ensemble. RKD filters malicious updates, forms a reliable benign ensemble, and distills their knowledge onto a refined global model using an unlabeled validation set and SWA, with optional exclusion or perturbation strategies for malicious clients. Extensive experiments on CIFAR-10, EMNIST, and Fashion-MNIST against adaptive attacks (A3FL, F3BA, DBA, ADBA, TSBA) show that RKD maintains high main-task accuracy while dramatically reducing attack success rates, outperforming state-of-the-art defenses in non-IID scenarios. The results highlight RKD's practical utility for securing FL in realistic, heterogeneous environments, with publicly available code enabling reproducibility and further exploration.

Abstract

Federated Learning (FL) enables collaborative model training across multiple devices while preserving data privacy. However, it remains susceptible to backdoor attacks, where malicious participants can compromise the global model. Existing defence methods are limited by strict assumptions on data heterogeneity (Non-Independent and Identically Distributed data) and the proportion of malicious clients, reducing their practicality and effectiveness. To overcome these limitations, we propose Robust Knowledge Distillation (RKD), a novel defence mechanism that enhances model integrity without relying on restrictive assumptions. RKD integrates clustering and model selection techniques to identify and filter out malicious updates, forming a reliable ensemble of models. It then employs knowledge distillation to transfer the collective insights from this ensemble to a global model. Extensive evaluations demonstrate that RKD effectively mitigates backdoor threats while maintaining high model performance, outperforming current state-of-the-art defence methods across various scenarios.

Figures (15)

• Figure 1: Performance of baselines and RKD on CIFAR-10 under Non-IID ($\alpha=0.3$), evaluated against $20\%$, $40\%$, and $60\%$ A3FL attacker clients.
• Figure 2: Performance of baselines and RKD on Fashion-MNIST under Non-IID ($\alpha=0.3$), evaluated against $20\%$, $40\%$, and $60\%$ A3FL attacker clients.
• Figure 3: Performance of baselines and RKD on CIFAR-10 under Non-IID ($\alpha=0.5$), evaluated against $20\%$, $40\%$, and $60\%$ F3BA attacker clients.
• Figure 4: Performance of baselines and RKD on EMNIST under Non-IID ($\alpha=0.5$), evaluated against $20\%$, $40\%$, and $60\%$ F3BA attacker clients.
• Figure 5: Performance of baselines and RKD on CIFAR-10 under Non-IID ($\alpha=0.9$), evaluated against $20\%$, $40\%$, and $60\%$ DBA attacker clients.