Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning

Jingyuan Zhang, Yiyang Duan, Shuaicheng Niu, Yang Cao, Wei Yang Bryan Lim

TL;DR

Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA.

Abstract

Federated Domain Adaptation (FDA) is a Federated Learning (FL) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging-based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client). To address this, we propose a novel framework called \textbf{M}ulti-domain \textbf{P}rototype-based \textbf{F}ederated Fine-\textbf{T}uning (MPFT). MPFT fine-tunes a pre-trained model using multi-domain prototypes, i.e., pretrained representations enriched with domain-specific information from category-specific local data. This enables supervised learning on the server to derive a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA. Notably, MPFT achieves convergence within a single communication round, greatly reducing computation and communication costs. To ensure privacy, MPFT applies differential privacy to protect the prototypes. Additionally, we develop a prototype-based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at \url{https://anonymous.4open.science/r/DomainFL/}.

Enhancing Federated Domain Adaptation with Multi-Domain Prototype-Based Federated Fine-Tuning

TL;DR

Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA.

Abstract

Federated Domain Adaptation (FDA) is a Federated Learning (FL) scenario where models are trained across multiple clients with unique data domains but a shared category space, without transmitting private data. The primary challenge in FDA is data heterogeneity, which causes significant divergences in gradient updates when using conventional averaging-based aggregation methods, reducing the efficacy of the global model. This further undermines both in-domain and out-of-domain performance (within the same federated system but outside the local client). To address this, we propose a novel framework called \textbf{M}ulti-domain \textbf{P}rototype-based \textbf{F}ederated Fine-\textbf{T}uning (MPFT). MPFT fine-tunes a pre-trained model using multi-domain prototypes, i.e., pretrained representations enriched with domain-specific information from category-specific local data. This enables supervised learning on the server to derive a globally optimized adapter that is subsequently distributed to local clients, without the intrusion of data privacy. Empirical results show that MPFT significantly improves both in-domain and out-of-domain accuracy over conventional methods, enhancing knowledge preservation and adaptation in FDA. Notably, MPFT achieves convergence within a single communication round, greatly reducing computation and communication costs. To ensure privacy, MPFT applies differential privacy to protect the prototypes. Additionally, we develop a prototype-based feature space hijacking attack to evaluate robustness, confirming that raw data samples remain unrecoverable even after extensive training epochs. The complete implementation of MPFL is available at \url{https://anonymous.4open.science/r/DomainFL/}.

Paper Structure

This paper contains 41 sections, 2 theorems, 27 equations, 9 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem statement
  4. Scenario assumption.
  5. Optimization goal.
  6. Evaluation metrics.
  7. Methodology
  8. Overview.
  9. Prototype Generation.
  10. Global adapter initialization.
  11. Few-shot local adaptation.
  12. Convergence.
  13. Experiment
  14. Datasets and Models.
  15. Implementation details.
  16. ...and 26 more sections

Key Result

Theorem 1

For a smooth, non-convex loss function $\mathcal{L}$ with a Lipschitz continuous gradient with constant $L$, the global fine-tuning using prototypes ${\mathbb{D}}^{\mathcal{P}}$ converges. The sequence of updates for the global adapter $A^{\mathcal{G}}$ achieves a monotonic decrease in the loss func where $c = \eta - \frac{L \eta^2}{2}$ is a positive constant, ensuring that the step size is approp

Figures (9)

  • Figure 1: Comparison of MPFT to centralized learning and previous averaging-based FL approaches.
  • Figure 2: An overview of MPFL.
  • Figure 3: Comparison of different FL methods across various DomainNet subset sizes.
  • Figure 4: Few-shot performance comparison of local adaptation with different KD weights.
  • Figure 5: t-SNE visualization of multiple datasets with their corresponding prototypes.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem 1: Convergence of fine-tuning with prototypes
  • Corollary 1.1: Convergence to stationary point and rate