Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaboration

Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Bo Gao, Quyang Pan, Tianliu He, Xuefeng Jiang

TL;DR

This work tackles the bottleneck of homogeneous models in hierarchical federated learning under End-Edge-Cloud Collaboration (EECC) by proposing Agglomerative Federated Learning (FedAgg). FedAgg enables tier-by-tier growth of model size and capability through Bridge Sample Based Online Distillation Protocol (BSBODP), allowing end devices to contribute to increasingly larger cloud-side models while preserving data privacy. Key contributions include a model-agnostic distillation mechanism across parent-child pairs, recursive agglomeration from leaf to cloud, and deployment-flexibility guarantees supported by formal interaction protocols. Empirical results on CIFAR-10 and CIFAR-100 demonstrate improved accuracy and faster convergence over state-of-the-art baselines, validating the practicality of training larger models in EECC settings.

Abstract

Federated Learning (FL) enables training Artificial Intelligence (AI) models over end devices without compromising their privacy. As computing tasks are increasingly performed by a combination of cloud, edge, and end devices, FL can benefit from this End-Edge-Cloud Collaboration (EECC) paradigm to achieve collaborative device-scale expansion with real-time access. Although Hierarchical Federated Learning (HFL) supports multi-tier model aggregation suitable for EECC, prior works assume the same model structure on all computing nodes, constraining the model scale by the weakest end devices. To address this issue, we propose Agglomerative Federated Learning (FedAgg), which is a novel EECC-empowered FL framework that allows the trained models from end, edge, to cloud to grow larger in size and stronger in generalization ability. FedAgg recursively organizes computing nodes among all tiers based on Bridge Sample Based Online Distillation Protocol (BSBODP), which enables every pair of parent-child computing nodes to mutually transfer and distill knowledge extracted from generated bridge samples. This design enhances the performance by exploiting the potential of larger models, with privacy constraints of FL and flexibility requirements of EECC both satisfied. Experiments under various settings demonstrate that FedAgg outperforms state-of-the-art methods by an average of 4.53\% accuracy gains and remarkable improvements in convergence rate.

Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaboration

TL;DR

This work tackles the bottleneck of homogeneous models in hierarchical federated learning under End-Edge-Cloud Collaboration (EECC) by proposing Agglomerative Federated Learning (FedAgg). FedAgg enables tier-by-tier growth of model size and capability through Bridge Sample Based Online Distillation Protocol (BSBODP), allowing end devices to contribute to increasingly larger cloud-side models while preserving data privacy. Key contributions include a model-agnostic distillation mechanism across parent-child pairs, recursive agglomeration from leaf to cloud, and deployment-flexibility guarantees supported by formal interaction protocols. Empirical results on CIFAR-10 and CIFAR-100 demonstrate improved accuracy and faster convergence over state-of-the-art baselines, validating the practicality of training larger models in EECC settings.

Abstract

Federated Learning (FL) enables training Artificial Intelligence (AI) models over end devices without compromising their privacy. As computing tasks are increasingly performed by a combination of cloud, edge, and end devices, FL can benefit from this End-Edge-Cloud Collaboration (EECC) paradigm to achieve collaborative device-scale expansion with real-time access. Although Hierarchical Federated Learning (HFL) supports multi-tier model aggregation suitable for EECC, prior works assume the same model structure on all computing nodes, constraining the model scale by the weakest end devices. To address this issue, we propose Agglomerative Federated Learning (FedAgg), which is a novel EECC-empowered FL framework that allows the trained models from end, edge, to cloud to grow larger in size and stronger in generalization ability. FedAgg recursively organizes computing nodes among all tiers based on Bridge Sample Based Online Distillation Protocol (BSBODP), which enables every pair of parent-child computing nodes to mutually transfer and distill knowledge extracted from generated bridge samples. This design enhances the performance by exploiting the potential of larger models, with privacy constraints of FL and flexibility requirements of EECC both satisfied. Experiments under various settings demonstrate that FedAgg outperforms state-of-the-art methods by an average of 4.53\% accuracy gains and remarkable improvements in convergence rate.
Paper Structure (26 sections, 22 equations, 5 figures, 5 tables, 2 algorithms)

This paper contains 26 sections, 22 equations, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hierarchical Federated Learning
  4. Training Larger Model in Federated Learning
  5. Preliminary
  6. End-Edge-Cloud Collaboration
  7. Hierarchical Federated Learning Empowered by End-Edge-Cloud Collaboration
  8. Interaction Protocols in Federated Learning
  9. Agglomerative Federated Learning
  10. Motivation and Overview
  11. Bridge Sample Based Online Distillation Protocol
  12. Recursive Agglomeration in End-Edge-Cloud Networks
  13. Deployment Flexibility Guarantees of Agglomerative Federated Learning
  14. Experiments
  15. Experiment Setup
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Framework of end-edge-cloud collaboration.
  • Figure 2: Expected model scales on different tiers in an end-edge-cloud network.
  • Figure 3: Overview of BSBODP. (1) Child Node Distillation on Bridge Samples. (2) Logits Extraction on Child Node. (3) Upload Logits to the Parent Node. (4) Parent Node Distillation on Bridge Samples. (5) Logits Extraction on Parent Node. (6) Distribute Logits to the Child Node.
  • Figure 4: Comparison of private samples and bridge samples.
  • Figure 5: Learning curves of the cloud-side model with varying numbers of devices and degrees of data heterogeneity. Results are obtained on CIFAR-10 dataset.