Multi-Modal Style Transfer-based Prompt Tuning for Efficient Federated Domain Generalization

TL;DR

This work addresses Federated Domain Generalization (FDG) by proposing FaST-PT, a framework that integrates CLIP-based multi-modal style transfer (MST) with a lightweight prompt-tuning strategy. MST augments embeddings by transferring style across external domains under text supervision, while a dual-prompt module separates global and domain-specific knowledge, further enhanced by domain-aware prompt generation to adapt to unseen domains. Experiments across PACS, VLCS, OfficeHome, and DomainNet show FaST-PT achieving state-of-the-art results and robust ablations validate the effectiveness and efficiency of MST and prompt components. The approach offers privacy-friendly, communication-efficient FDG with strong generalization to unseen domains, leveraging a unified CLIP backbone and lightweight prompt exchanges.

Abstract

Federated Domain Generalization (FDG) aims to collaboratively train a global model across distributed clients that can generalize well on unseen domains. However, existing FDG methods typically struggle with cross-client data heterogeneity and incur significant communication and computation overhead. To address these challenges, this paper presents a new FDG framework, dubbed FaST-PT, which facilitates local feature augmentation and efficient unseen domain adaptation in a distributed manner. First, we propose a lightweight Multi-Modal Style Transfer (MST) method to transform image embedding under text supervision, which could expand the training data distribution and mitigate domain shift. We then design a dual-prompt module that decomposes the prompt into global and domain prompts. Specifically, global prompts capture general knowledge from augmented embedding across clients, while domain prompts capture domain-specific knowledge from local data. Besides, Domain-aware Prompt Generation (DPG) is introduced to adaptively generate suitable prompts for each sample, which facilitates unseen domain adaptation through knowledge fusion. Extensive experiments on four cross-domain benchmark datasets, e.g., PACS and DomainNet, demonstrate the superior performance of FaST-PT over SOTA FDG methods such as FedDG-GA and DiPrompt. Ablation studies further validate the effectiveness and efficiency of FaST-PT.

Figures (4)

• Figure 1: Framework of our proposed FaST-PT. The left side illustrates the local training process, while the right side depicts the communication between client and server. The client-side training is divided into two phases: (a) MST and (b) FPT. (a) In the MST phase, the client employs a frozen CLIP model along with text descriptions from various domains to train a transform network, enabling style transfer at the embedding level. (b) In the FPT phase, the client trains global prompts, domain prompts, and domain classifiers using the augmented data. The inference process is also depicted on the server side.
• Figure 2: Nearest Neighbors on PACS and DomainNet.
• Figure 3: Effect of prompt length on PACS and DomainNet datasets over three runs with different random seeds.
• Figure 4: Few-shot performance on PACS and DomainNet datasets over three runs with different random seeds.