Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning

Jiaqi Wang, Chenxu Zhao, Lingjuan Lyu, Quanzeng You, Mengdi Huai, Fenglong Ma

TL;DR

FedType addresses federated learning with heterogeneous client models by introducing identical small proxy models and a bidirectional uncertainty-based reciprocity framework to transfer knowledge without public data. The core mechanisms—forward knowledge distillation from large private models to proxies and backward knowledge distillation guided by dynamic conformal prediction-based uncertainty sets with consensus weighting—enable effective model aggregation while preserving privacy and reducing communication. Empirical results across cross-device and cross-silo settings on multiple datasets demonstrate consistent performance gains for the private and proxy models over global baselines, validating the approach's practicality and robustness. The work offers a data-free, communication-efficient solution for heterogenous FL with potential impact in privacy-sensitive domains such as healthcare and finance, while leaving room for improvements in proxy-model selection and computational efficiency.

Abstract

This paper presents FedType, a simple yet pioneering framework designed to fill research gaps in heterogeneous model aggregation within federated learning (FL). FedType introduces small identical proxy models for clients, serving as agents for information exchange, ensuring model security, and achieving efficient communication simultaneously. To transfer knowledge between large private and small proxy models on clients, we propose a novel uncertainty-based asymmetrical reciprocity learning method, eliminating the need for any public data. Comprehensive experiments conducted on benchmark datasets demonstrate the efficacy and generalization ability of FedType across diverse settings. Our approach redefines federated learning paradigms by bridging model heterogeneity, eliminating reliance on public data, prioritizing client privacy, and reducing communication costs.

Bridging Model Heterogeneity in Federated Learning via Uncertainty-based Asymmetrical Reciprocity Learning

TL;DR

FedType addresses federated learning with heterogeneous client models by introducing identical small proxy models and a bidirectional uncertainty-based reciprocity framework to transfer knowledge without public data. The core mechanisms—forward knowledge distillation from large private models to proxies and backward knowledge distillation guided by dynamic conformal prediction-based uncertainty sets with consensus weighting—enable effective model aggregation while preserving privacy and reducing communication. Empirical results across cross-device and cross-silo settings on multiple datasets demonstrate consistent performance gains for the private and proxy models over global baselines, validating the approach's practicality and robustness. The work offers a data-free, communication-efficient solution for heterogenous FL with potential impact in privacy-sensitive domains such as healthcare and finance, while leaving room for improvements in proxy-model selection and computational efficiency.

Abstract

This paper presents FedType, a simple yet pioneering framework designed to fill research gaps in heterogeneous model aggregation within federated learning (FL). FedType introduces small identical proxy models for clients, serving as agents for information exchange, ensuring model security, and achieving efficient communication simultaneously. To transfer knowledge between large private and small proxy models on clients, we propose a novel uncertainty-based asymmetrical reciprocity learning method, eliminating the need for any public data. Comprehensive experiments conducted on benchmark datasets demonstrate the efficacy and generalization ability of FedType across diverse settings. Our approach redefines federated learning paradigms by bridging model heterogeneity, eliminating reliance on public data, prioritizing client privacy, and reducing communication costs.
Paper Structure (41 sections, 8 equations, 8 figures, 9 tables, 1 algorithm)

This paper contains 41 sections, 8 equations, 8 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Heterogeneous Federated Learning
  4. Conformal Prediction
  5. Methodology
  6. Local Update via Uncertainty-based Asymmetrical Reciprocity Learning
  7. Uncertainty-based Behavior Imitation Learning
  8. Backward Knowledge Distillation Loss
  9. Conformal Model Training
  10. Uncertainty Set Prediction
  11. Consensus Weight Estimation
  12. Server Update
  13. Experiments
  14. Experimental Setups
  15. Results of the Heterogeneous Model Setting
  16. ...and 26 more sections

Figures (8)

  • Figure 1: Overview of the proposed FedType framework. (a) demonstrates the workflow of the proposed FedType to address the model-heterogeneous issue in FL, (b) is the local update demonstration for a data sample $\mathbf{x}_j$ using the proposed uncertainty-based asymmetrical reciprocity learning, and (c) is the illustration of backward knowledge distillation with the proposed uncertainty-based behavior imitation learning.
  • Figure 2: Results (%) of the cross-silo evaluation.
  • Figure 3: Calibration function $g(\Delta^t, \lambda)$ study.
  • Figure 4: Homogenenous evaluation.
  • Figure 5: The empirical convergence of our proposed FedType.
  • ...and 3 more figures