Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Heterogeneous Federated Learning with Splited Language Model

Yifan Shi, Yuhui Zhang, Ziyue Huang, Xiaofeng Yang, Li Shen, Wei Chen, Xueqian Wang

TL;DR

This paper addresses the challenge of training large transformer models in federated settings with data heterogeneity and limited device resources. It proposes a split-learning approach where a Pre-trained Image Transformer (PIT) encoder resides on the server while client-side heads and tails are trained locally, coupled with two algorithms: FedV, which aggregates exact server gradients, and FedVZ, which uses zeroth-order gradient estimates to preserve privacy in black-box scenarios. The authors systematically evaluate PIT-based FSL across CIFAR-10/100 and Tiny-ImageNet using multiple PIT families under non-IID Dirichlet and Pathological partitions, showing superior accuracy and convergence over strong baselines such as FedAvg, Fed-RoD, Ditto, and FESTA, with FedVZ offering privacy advantages. The work demonstrates that PIT-enabled FSL can substantially reduce training overhead on edge devices while maintaining robustness to data heterogeneity, providing a practical blueprint for privacy-preserving distributed learning with transformer models.

Abstract

Federated Split Learning (FSL) is a promising distributed learning paradigm in practice, which gathers the strengths of both Federated Learning (FL) and Split Learning (SL) paradigms, to ensure model privacy while diminishing the resource overhead of each client, especially on large transformer models in a resource-constrained environment, e.g., Internet of Things (IoT). However, almost all works merely investigate the performance with simple neural network models in FSL. Despite the minor efforts focusing on incorporating Vision Transformers (ViT) as model architectures, they train ViT from scratch, thereby leading to enormous training overhead in each device with limited resources. Therefore, in this paper, we harness Pre-trained Image Transformers (PITs) as the initial model, coined FedV, to accelerate the training process and improve model robustness. Furthermore, we propose FedVZ to hinder the gradient inversion attack, especially having the capability compatible with black-box scenarios, where the gradient information is unavailable. Concretely, FedVZ approximates the server gradient by utilizing a zeroth-order (ZO) optimization, which replaces the backward propagation with just one forward process. Empirically, we are the first to provide a systematic evaluation of FSL methods with PITs in real-world datasets, different partial device participations, and heterogeneous data splits. Our experiments verify the effectiveness of our algorithms.

Heterogeneous Federated Learning with Splited Language Model

TL;DR

This paper addresses the challenge of training large transformer models in federated settings with data heterogeneity and limited device resources. It proposes a split-learning approach where a Pre-trained Image Transformer (PIT) encoder resides on the server while client-side heads and tails are trained locally, coupled with two algorithms: FedV, which aggregates exact server gradients, and FedVZ, which uses zeroth-order gradient estimates to preserve privacy in black-box scenarios. The authors systematically evaluate PIT-based FSL across CIFAR-10/100 and Tiny-ImageNet using multiple PIT families under non-IID Dirichlet and Pathological partitions, showing superior accuracy and convergence over strong baselines such as FedAvg, Fed-RoD, Ditto, and FESTA, with FedVZ offering privacy advantages. The work demonstrates that PIT-enabled FSL can substantially reduce training overhead on edge devices while maintaining robustness to data heterogeneity, providing a practical blueprint for privacy-preserving distributed learning with transformer models.

Abstract

Federated Split Learning (FSL) is a promising distributed learning paradigm in practice, which gathers the strengths of both Federated Learning (FL) and Split Learning (SL) paradigms, to ensure model privacy while diminishing the resource overhead of each client, especially on large transformer models in a resource-constrained environment, e.g., Internet of Things (IoT). However, almost all works merely investigate the performance with simple neural network models in FSL. Despite the minor efforts focusing on incorporating Vision Transformers (ViT) as model architectures, they train ViT from scratch, thereby leading to enormous training overhead in each device with limited resources. Therefore, in this paper, we harness Pre-trained Image Transformers (PITs) as the initial model, coined FedV, to accelerate the training process and improve model robustness. Furthermore, we propose FedVZ to hinder the gradient inversion attack, especially having the capability compatible with black-box scenarios, where the gradient information is unavailable. Concretely, FedVZ approximates the server gradient by utilizing a zeroth-order (ZO) optimization, which replaces the backward propagation with just one forward process. Empirically, we are the first to provide a systematic evaluation of FSL methods with PITs in real-world datasets, different partial device participations, and heterogeneous data splits. Our experiments verify the effectiveness of our algorithms.
Paper Structure (15 sections, 5 equations, 3 figures, 5 tables, 1 algorithm)

This paper contains 15 sections, 5 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Problem Setup
  5. Algorithmic Details
  6. Experiments
  7. Experimental Setup
  8. Performance Evaluation
  9. Ablation Study
  10. Conclusion
  11. More Details in the Implementation.
  12. Datasets and Data Partition
  13. More details about the FedV and FedVZ
  14. More Details about Baselines
  15. More Details in the Experiment.

Figures (3)

  • Figure 1: The averaged testing accuracy on CIFAR-10 and CIFAR-100 dataset for all compared methods.
  • Figure 2: Effect of the main hyper-parameters: local iteration steps $K$, the number of participated clients $M$, and the local model aggregation interval: fedround.
  • Figure 3: Tiny-ImageNet