Feature-based Federated Transfer Learning: Communication Efficiency, Robustness and Privacy

TL;DR

This work tackles the communication bottlenecks of federated learning by introducing feature-based federated transfer learning (FbFTL), which uploads extracted features and outputs instead of gradients to drastically reduce uplink traffic. FbFTL partitions a pre-trained source model, fixes the feature extractor, and trains only a task-specific sub-model, enabling order-of-magnitude reductions in uplink and downlink payloads while shifting computation toward the server. The paper provides rigorous payload, time-complexity, robustness, and privacy analyses, including new privacy notions for label privacy leakage and feature privacy leakage, and validates the approach on image (VGG-16 CIFAR-10) and NLP (FLAN-T5-small) tasks with differential privacy considerations. The results demonstrate that FbFTL achieves comparable accuracy to gradient-based baselines under much smaller communication budgets, improved resilience to packet loss, and practical privacy protections, highlighting its potential for deployment in bandwidth-constrained, privacy-sensitive federated settings.

Abstract

In this paper, we propose feature-based federated transfer learning as a novel approach to improve communication efficiency by reducing the uplink payload by multiple orders of magnitude compared to that of existing approaches in federated learning and federated transfer learning. Specifically, in the proposed feature-based federated learning, we design the extracted features and outputs to be uploaded instead of parameter updates. For this distributed learning model, we determine the required payload and provide comparisons with the existing schemes. Subsequently, we analyze the robustness of feature-based federated transfer learning against packet loss, data insufficiency, and quantization. Finally, we address privacy considerations by defining and analyzing label privacy leakage and feature privacy leakage, and investigating mitigating approaches. For all aforementioned analyses, we evaluate the performance of the proposed learning scheme via experiments on an image classification task and a natural language processing task to demonstrate its effectiveness.

Figures (12)

• Figure 1: Diagram of the iterative training process of model-based federated transfer learning.
• Figure 2: Diagram of the training process of feature-based federated transfer learning, which reduces to one-time communication of the output and the intermediate output (i.e., extracted features).
• Figure 3: ImageNet samples with labels.
• Figure 4: CIFAR-10 samples with labels.
• Figure 5: Diagram of the VGG-16 model for training on CIFAR-10 dataset. "Conv $\{$receptive field size$\}$ - $\{$number of output channels$\}$" depicts the convolutional layers. "FC $\{$output size$\}$" depicts the fully connected layers.

Theorems & Definitions (6)

• Definition 1
• Lemma 1
• proof
• Definition 2
• Lemma 2
• proof