Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning

Yubin Zheng, Pak-Hei Yeung, Jing Xia, Tianjie Ju, Peng Tang, Weidong Qiu, Jagath C. Rajapakse

TL;DR

This work addresses domain shift and label heterogeneity in federated learning by adapting CLIP through a novel FedDEAP framework. It introduces ETF-constrained semantic and domain transformation networks to disentangle image features, and a dual-prompt design with a global semantic prompt and a local domain prompt to preserve shared semantics while capturing domain-specific cues. The approach is theoretically motivated by mutual-information considerations and empirically validated across four datasets, showing state-of-the-art performance and robustness to label imbalance, with favorable efficiency. FedDEAP thus offers a principled, scalable method for cross-domain CLIP fine-tuning in federated environments, enabling improved generalization and practical deployment.

Abstract

Federated learning (FL) enables multiple clients to collaboratively train machine learning models without exposing local data, balancing performance and privacy. However, domain shift and label heterogeneity across clients often hinder the generalization of the aggregated global model. Recently, large-scale vision-language models like CLIP have shown strong zero-shot classification capabilities, raising the question of how to effectively fine-tune CLIP across domains in a federated setting. In this work, we propose an adaptive federated prompt tuning framework, FedDEAP, to enhance CLIP's generalization in multi-domain scenarios. Our method includes the following three key components: (1) To mitigate the loss of domain-specific information caused by label-supervised tuning, we disentangle semantic and domain-specific features in images by using semantic and domain transformation networks with unbiased mappings; (2) To preserve domain-specific knowledge during global prompt aggregation, we introduce a dual-prompt design with a global semantic prompt and a local domain prompt to balance shared and personalized information; (3) To maximize the inclusion of semantic and domain information from images in the generated text features, we align textual and visual representations under the two learned transformations to preserve semantic and domain consistency. Theoretical analysis and extensive experiments on four datasets demonstrate the effectiveness of our method in enhancing the generalization of CLIP for federated image recognition across multiple domains.

FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning

TL;DR

This work addresses domain shift and label heterogeneity in federated learning by adapting CLIP through a novel FedDEAP framework. It introduces ETF-constrained semantic and domain transformation networks to disentangle image features, and a dual-prompt design with a global semantic prompt and a local domain prompt to preserve shared semantics while capturing domain-specific cues. The approach is theoretically motivated by mutual-information considerations and empirically validated across four datasets, showing state-of-the-art performance and robustness to label imbalance, with favorable efficiency. FedDEAP thus offers a principled, scalable method for cross-domain CLIP fine-tuning in federated environments, enabling improved generalization and practical deployment.

Abstract

Federated learning (FL) enables multiple clients to collaboratively train machine learning models without exposing local data, balancing performance and privacy. However, domain shift and label heterogeneity across clients often hinder the generalization of the aggregated global model. Recently, large-scale vision-language models like CLIP have shown strong zero-shot classification capabilities, raising the question of how to effectively fine-tune CLIP across domains in a federated setting. In this work, we propose an adaptive federated prompt tuning framework, FedDEAP, to enhance CLIP's generalization in multi-domain scenarios. Our method includes the following three key components: (1) To mitigate the loss of domain-specific information caused by label-supervised tuning, we disentangle semantic and domain-specific features in images by using semantic and domain transformation networks with unbiased mappings; (2) To preserve domain-specific knowledge during global prompt aggregation, we introduce a dual-prompt design with a global semantic prompt and a local domain prompt to balance shared and personalized information; (3) To maximize the inclusion of semantic and domain information from images in the generated text features, we align textual and visual representations under the two learned transformations to preserve semantic and domain consistency. Theoretical analysis and extensive experiments on four datasets demonstrate the effectiveness of our method in enhancing the generalization of CLIP for federated image recognition across multiple domains.
Paper Structure (22 sections, 30 equations, 7 figures, 5 tables)

This paper contains 22 sections, 30 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning
  4. CLIP and Prompt Tuning
  5. Federated Fine-tuning for CLIP
  6. Methods
  7. Problem Definition
  8. Semantic and Domain Transformation Networks Training with ETF Classifier
  9. Global Shared Semantic and Local Personalized Domain Prompts Training
  10. Mutual Information Preservation Analysis
  11. Experiments
  12. Experimental Settings
  13. Main Results
  14. Performance under Category Imbalance
  15. Domain Adaptivity of Prompts
  16. ...and 7 more sections

Figures (7)

  • Figure 1: The framework of the proposed FedDEAP method. Each client decouples semantic and domain features from images using transformation networks (TNs) guided by global semantic and domain ETF, and aligns the prompt with the image in both feature spaces. The global prompt and TNs are aggregated on the server at the end of each round.
  • Figure 2: Average classification accuracy across domains under different Dirichlet $\alpha$ values on the PACS and DomainNet datasets.
  • Figure 3: Classification accuracy heatmaps using prompts from different domains on various image domains in PACS and DomainNet.
  • Figure 4: Visualization of prompt and image features in the embedding spaces of (a) FedDEAP and (b) PromptFL.
  • Figure 5: Grad-CAM analysis of semantic and domain transformation networks across different domains and categories.
  • ...and 2 more figures