Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data

Ji Liu, Juncheng Jia, Hong Zhang, Yuhui Yun, Leye Wang, Yang Zhou, Huaiyu Dai, Dejing Dou

TL;DR

FedDUMAP tackles the inefficiency and resource constraints of federated learning by jointly leveraging shared server data and distributed device data. It introduces a dynamic server update (FedDU), momentum-based optimization (FedDUM) that decouples server and device momentum, and a layer-adaptive pruning mechanism (FedAP) to cut computational cost while preserving accuracy. The framework quantifies non-IID distributions with Jensen–Shannon divergence and balances contributions via normalized server gradients and adaptive steps. Extensive experiments on CIFAR-10/100 across multiple architectures show substantial gains in accuracy (up to 20.4%), speed (up to 16.9x), and cost reductions (up to 62.6%) over strong baselines, highlighting the practical impact for scalable, privacy-conscious FL.

Abstract

Despite achieving remarkable performance, Federated Learning (FL) encounters two important problems, i.e., low training efficiency and limited computational resources. In this paper, we propose a new FL framework, i.e., FedDUMAP, with three original contributions, to leverage the shared insensitive data on the server in addition to the distributed data in edge devices so as to efficiently train a global model. First, we propose a simple dynamic server update algorithm, which takes advantage of the shared insensitive data on the server while dynamically adjusting the update steps on the server in order to speed up the convergence and improve the accuracy. Second, we propose an adaptive optimization method with the dynamic server update algorithm to exploit the global momentum on the server and each local device for superior accuracy. Third, we develop a layer-adaptive model pruning method to carry out specific pruning operations, which is adapted to the diverse features of each layer so as to attain an excellent trade-off between effectiveness and efficiency. Our proposed FL model, FedDUMAP, combines the three original techniques and has a significantly better performance compared with baseline approaches in terms of efficiency (up to 16.9 times faster), accuracy (up to 20.4% higher), and computational cost (up to 62.6% smaller).

Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data

TL;DR

FedDUMAP tackles the inefficiency and resource constraints of federated learning by jointly leveraging shared server data and distributed device data. It introduces a dynamic server update (FedDU), momentum-based optimization (FedDUM) that decouples server and device momentum, and a layer-adaptive pruning mechanism (FedAP) to cut computational cost while preserving accuracy. The framework quantifies non-IID distributions with Jensen–Shannon divergence and balances contributions via normalized server gradients and adaptive steps. Extensive experiments on CIFAR-10/100 across multiple architectures show substantial gains in accuracy (up to 20.4%), speed (up to 16.9x), and cost reductions (up to 62.6%) over strong baselines, highlighting the practical impact for scalable, privacy-conscious FL.

Abstract

Despite achieving remarkable performance, Federated Learning (FL) encounters two important problems, i.e., low training efficiency and limited computational resources. In this paper, we propose a new FL framework, i.e., FedDUMAP, with three original contributions, to leverage the shared insensitive data on the server in addition to the distributed data in edge devices so as to efficiently train a global model. First, we propose a simple dynamic server update algorithm, which takes advantage of the shared insensitive data on the server while dynamically adjusting the update steps on the server in order to speed up the convergence and improve the accuracy. Second, we propose an adaptive optimization method with the dynamic server update algorithm to exploit the global momentum on the server and each local device for superior accuracy. Third, we develop a layer-adaptive model pruning method to carry out specific pruning operations, which is adapted to the diverse features of each layer so as to attain an excellent trade-off between effectiveness and efficiency. Our proposed FL model, FedDUMAP, combines the three original techniques and has a significantly better performance compared with baseline approaches in terms of efficiency (up to 16.9 times faster), accuracy (up to 20.4% higher), and computational cost (up to 62.6% smaller).
Paper Structure (16 sections, 1 theorem, 13 equations, 17 figures, 13 tables, 3 algorithms)

This paper contains 16 sections, 1 theorem, 13 equations, 17 figures, 13 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. System Model
  5. Server Update
  6. Momentum-based Optimization
  7. Adaptive Pruning
  8. Experiments
  9. Experimental Setup
  10. Evaluation with non-IID Data
  11. Evaluation on FedDU
  12. Evaluation on FedDUM
  13. Evaluation on FedAP
  14. Evaluation on FedDUMAP
  15. Ablation Study
  16. ...and 1 more sections

Key Result

Theorem 3.1

Local momentum deviates from the centralized one at linear rate $O(e^{\lambda^+E})$.

Figures (17)

  • Figure 1: The training process of FedDUMAP Framework.
  • Figure 2: The accuracy of FedDU with diverse amounts of server data on CIFAR-10.$1\%$, $5\%$, $10\%$ represent the value of $p$ (see details in Section 4.1).
  • Figure 3: The accuracy and training time with diverse model update methods corresponding to FedDU with $p = 5\%$ and $p = 10\%$. CNN is with CIFAR-10 and ResNet is with CIFAR-100.
  • Figure 4: The accuracy and training time with diverse model update methods corresponding to FedDU based on CIFAR-10.
  • Figure 5: The accuracy and training time with diverse model update methods corresponding to FedDU based on CIFAR-100.
  • ...and 12 more figures

Theorems & Definitions (1)

  • Theorem 3.1