FedQP: Towards Accurate Federated Learning using Quadratic Programming Guided Mutation

TL;DR

A novel mutation-based FL approach named FedQP is proposed, utilizing a quadratic programming strategy to regulate the mutation directions wisely, which can effectively guide the model to optimize towards a well-generalized area (i.e., flat area).

Abstract

Due to the advantages of privacy-preserving, Federated Learning (FL) is widely used in distributed machine learning systems. However, existing FL methods suffer from low-inference performance caused by data heterogeneity. Specifically, due to heterogeneous data, the optimization directions of different local models vary greatly, making it difficult for the traditional FL method to get a generalized global model that performs well on all clients. As one of the state-of-the-art FL methods, the mutation-based FL method attempts to adopt a stochastic mutation strategy to guide the model training towards a well-generalized area (i.e., flat area in the loss landscape). Specifically, mutation allows the model to shift within the solution space, providing an opportunity to escape areas with poor generalization (i.e., sharp area). However, the stochastic mutation strategy easily results in diverse optimal directions of mutated models, which limits the performance of the existing mutation-based FL method. To achieve higher performance, this paper proposes a novel mutation-based FL approach named FedQP, utilizing a quadratic programming strategy to regulate the mutation directions wisely. By biasing the model mutation towards the direction of gradient update rather than traditional random mutation, FedQP can effectively guide the model to optimize towards a well-generalized area (i.e., flat area). Experiments on multiple well-known datasets show that our quadratic programming-guided mutation strategy effectively improves the inference accuracy of the global model in various heterogeneous data scenarios.

Figures (4)

• Figure 1: Motivation of mutation range.
• Figure 2: Framework and workflow of our approach.
• Figure 3: Learning curves of our method and all the baseline methods on CIFAR-10 with ResNet-18.
• Figure 4: Ablation study results.