Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning

TL;DR

A federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification with powerful foundation models, like CLIP, to adaptively unlock specific domain knowledge for each test sample in order to provide them with personalized prompts.

Abstract

Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. Previous researches often require training the complete model parameters. However, the emergence of powerful pre-trained models makes it possible to achieve higher performance with fewer learnable parameters in FL. In this paper, we propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification with powerful foundation models, like CLIP. Compared with direct federated prompt tuning, our core idea is to adaptively unlock specific domain knowledge for each test sample in order to provide them with personalized prompts. To implement this idea, we design an adaptive prompt tuning module, which consists of a meta prompt, an adaptive network, and some keys. The server randomly generates a set of keys and assigns a unique key to each client. Then all clients cooperatively train the global adaptive network and meta prompt with the local datasets and the frozen keys. Ultimately, the global aggregation model can assign a personalized prompt to CLIP based on the domain features of each test sample. We perform extensive experiments on two multi-domain image classification datasets across two different settings -- supervised and unsupervised. The results show that FedAPT can achieve better performance with less than 10\% of the number of parameters of the fully trained model, and the global model can perform well in diverse client domains simultaneously. The source code is available at \url{https://github.com/leondada/FedAPT}.

Figures (6)

• Figure 1: The impact of different prompts on performance in multi-domain scenarios. The first three prompts are manually set. One can see that when domain information is included in the prompt (in red), the accuracy of the corresponding domain is improved (red arrow), but that of other domains is decreased (green arrow). The bottom three rows are the learned prompts. On one hand, the prompts learned only in one domain perform significantly worse in other domains. On the other hand, we can obtain prompts through FedAPT that are applicable across all domains.
• Figure 2: Overview of FedAPT. The federated training process is as follows: Step1: The server first randomly generates a set of random keys and assigns each client a frozen key. Step2: In local training, the key is used to give a constraint to the prompt. In addition, each client trains the adaptive network $\mathcal{Q}_n$ with local data. Step3: After local training, the server averages the prompts and adaptive networks learned by the clients and sends the results back to the clients. Repeat Steps 2$\sim$3 for multiple rounds, we can obtain $\mathbf{p}_g$ and $\mathcal{Q}_g$ to establish our target global model which can generate a personalized prompt according to image features.
• Figure 3: Improve inference efficiency.
• Figure 4: (a) Office-Caltech10, which includes four domains. (b) DomainNet, which includes six domains.
• Figure 5: Unbalanced label distribution controlled by $\beta$. When $\beta$ is small, there is almost no category overlap between different clients.