Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning

Huy Q. Le, Minh N. H. Nguyen, Shashi Raj Pandey, Chaoning Zhang, Choong Seon Hong

TL;DR

This work demonstrates that the knowledge transfer mechanism achieves significant speedups and high personalized performance of local models and develops a novel knowledge distillation-based approach to study the extent of knowledge transfer between the global model and local models.

Abstract

In a practical setting, how to enable robust Federated Learning (FL) systems, both in terms of generalization and personalization abilities, is one important research question. It is a challenging issue due to the consequences of non-i.i.d. properties of client's data, often referred to as statistical heterogeneity, and small local data samples from the various data distributions. Therefore, to develop robust generalized global and personalized models, conventional FL methods need to redesign the knowledge aggregation from biased local models while considering huge divergence of learning parameters due to skewed client data. In this work, we demonstrate that the knowledge transfer mechanism achieves these objectives and develop a novel knowledge distillation-based approach to study the extent of knowledge transfer between the global model and local models. Henceforth, our method considers the suitability of transferring the outcome distribution and (or) the embedding vector of representation from trained models during cross-device knowledge transfer using a small proxy dataset in heterogeneous FL. In doing so, we alternatively perform cross-device knowledge transfer following general formulations as 1) global knowledge transfer and 2) on-device knowledge transfer. Through simulations on three federated datasets, we show the proposed method achieves significant speedups and high personalized performance of local models. Furthermore, the proposed approach offers a more stable algorithm than other baselines during the training, with minimal communication data load when exchanging the trained model's outcomes and representation.

CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning

TL;DR

This work demonstrates that the knowledge transfer mechanism achieves significant speedups and high personalized performance of local models and develops a novel knowledge distillation-based approach to study the extent of knowledge transfer between the global model and local models.

Abstract

In a practical setting, how to enable robust Federated Learning (FL) systems, both in terms of generalization and personalization abilities, is one important research question. It is a challenging issue due to the consequences of non-i.i.d. properties of client's data, often referred to as statistical heterogeneity, and small local data samples from the various data distributions. Therefore, to develop robust generalized global and personalized models, conventional FL methods need to redesign the knowledge aggregation from biased local models while considering huge divergence of learning parameters due to skewed client data. In this work, we demonstrate that the knowledge transfer mechanism achieves these objectives and develop a novel knowledge distillation-based approach to study the extent of knowledge transfer between the global model and local models. Henceforth, our method considers the suitability of transferring the outcome distribution and (or) the embedding vector of representation from trained models during cross-device knowledge transfer using a small proxy dataset in heterogeneous FL. In doing so, we alternatively perform cross-device knowledge transfer following general formulations as 1) global knowledge transfer and 2) on-device knowledge transfer. Through simulations on three federated datasets, we show the proposed method achieves significant speedups and high personalized performance of local models. Furthermore, the proposed approach offers a more stable algorithm than other baselines during the training, with minimal communication data load when exchanging the trained model's outcomes and representation.
Paper Structure (17 sections, 10 equations, 23 figures, 11 tables, 1 algorithm)

This paper contains 17 sections, 10 equations, 23 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Cross-Device Knowledge Transfer in Federated Learning
  4. Generalized Model Construction
  5. On-device Learning
  6. Experiments
  7. Setting
  8. Experimental Results
  9. Conclusion
  10. Supplemental Results
  11. Cross Entropy Loss:
  12. CIFAR-10 and CIFAR-100 results
  13. Model Architecture
  14. Data Partition
  15. Detailed of C-Gen and C-Spec performance
  16. ...and 2 more sections

Figures (23)

  • Figure 1: An overview of Cross-Device Knowledge Transfer (CDKT) scheme in FL, integrating two mechanisms: 1) Generalized Model Construction for transferring knowledge from client models to the global model, and 2) On-Device Learning for transferring the generalized knowledge of global model to client models.
  • Figure 2: Comparative Analysis of $\textsf{CDKT-FL}$ under various settings against other baselines using the Fashion-MNIST dataset in Fixed Users scenario.
  • Figure 3: Comparative Analysis of $\textsf{CDKT-FL}$ under various settings against other baselines using the Fashion-MNIST dataset in Subset of Users scenario.
  • Figure 4: Effects of trade-off parameters in $\textsf{CDKT-FL}$ with Fashion-MNIST dataset in Fixed Users scenario.
  • Figure 5: Effects of trade-off parameters in $\textsf{CDKT-FL}$ with Fashion-MNIST dataset in Subset of Users scenario.
  • ...and 18 more figures