Adaptive Federated Learning Defences via Trust-Aware Deep Q-Networks

TL;DR

Federated learning faces poisoning and backdoor risks under partial observability. The authors propose a trust-aware Deep Q-Network defense framed as a partially observable Markov decision process, integrating multi-signal anomaly evidence with Bayesian belief tracking to adaptively weight client updates. The approach outperforms static defenses and other RL baselines on CIFAR-10, with accuracy improving while attack resistance remains controlled, aided by sequential belief updates that stabilize trust decisions. This work demonstrates a practical, reproducible method for robust FL defenses that leverages temporal evidence and partial observability to defend against adaptive adversaries.

Abstract

Federated learning is vulnerable to poisoning and backdoor attacks under partial observability. We formulate defence as a partially observable sequential decision problem and introduce a trust-aware Deep Q-Network that integrates multi-signal evidence into client trust updates while optimizing a long-horizon robustness--accuracy objective. On CIFAR-10, we (i) establish a baseline showing steadily improving accuracy, (ii) show through a Dirichlet sweep that increased client overlap consistently improves accuracy and reduces ASR with stable detection, and (iii) demonstrate in a signal-budget study that accuracy remains steady while ASR increases and ROC-AUC declines as observability is reduced, which highlights that sequential belief updates mitigate weaker signals. Finally, a comparison with random, linear-Q, and policy gradient controllers confirms that DQN achieves the best robustness--accuracy trade-off.

Figures (10)

• Figure 1: Trust-aware RL defence under partial observability. Client updates are converted to anomaly signals: alignment, $\lVert g\rVert$ magnitude deviation, and validation impact. This feeds to a Bayesian belief state update module. The DQN policy then acts on these beliefs by choosing, for each client, to increase, decrease, or hold its trust score, producing $TS_{t+1}$. These trust scores parameterize aggregation (reweighting/gating of updates) to form the global model update, and a reward based on accuracy and ASR trains the policy over rounds under secure aggregation.
• Figure 2: Baseline learning dynamics. (a) Test accuracy over rounds rises steadily, reaching $65.32\%$ by round 50. (b) Per-client belief trajectories separate early and then stabilize, indicating the agent’s ability to track client reliability under partial observability. (c) Final trust scores at round 50 (mean-normalized) showing low weights for suspected clients while preserving high trust for consistently benign participants.
• Figure 3: Agent performance over training rounds. We compare Linear-Q, DQN, Random, and Policy Gradient under a fixed FL setup. (a) shows test accuracy, where DQN steadily improves over rounds and attains substantially higher values than the other agents, which remain near chance. (b) shows backdoor ASR, with DQN exhibiting relatively lower and more stable levels, while the baselines fluctuate with higher variance. (c) reports ROC–AUC, where all agents trend downward with training but DQN maintains competitive performance. Overall, the results suggest that DQN achieves stronger accuracy while balancing robustness compared to simpler agents.
• Figure 4: Dirichlet heterogeneity sweep. Increasing $\alpha$ improves clean accuracy and reduces ASR, and variability narrows as $\alpha$ grows. ROC-AUC remains largely stable across $\alpha$ with only a mild downward drift at higher $\alpha$. (a) Accuracy, (b) ASR, (c) ROC-AUC.
• Figure 5: Signal-budget ablation (Full / No-Validation / Directional-only). Clean accuracy remains nearly unchanged, while robustness degrades as evidence is removed: ASR increases and ROC-AUC drops from $0.55$ to $0.48$. Bars show means over five seeds.