pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models

TL;DR

This work proposes pFedMMA, the first personalized federated learning framework that leverages multi-modal adapters for vision-language tasks, and achieves state-of-the-art trade-offs between personalization and generalization, outperforming recent federated prompt tuning methods.

Abstract

Vision-Language Models (VLMs) like CLIP have demonstrated remarkable generalization in zero- and few-shot settings, but adapting them efficiently to decentralized, heterogeneous data remains a challenge. While prompt tuning has emerged as a popular parameter-efficient approach in personalized federated learning, existing methods often sacrifice generalization in favor of personalization, struggling particularly on unseen classes or domains. In this work, we propose pFedMMA, the first personalized federated learning framework that leverages multi-modal adapters for vision-language tasks. Each adapter contains modality-specific up- and down-projection layers alongside a globally shared projection that aligns cross-modal features. Our optimization strategy allows clients to locally adapt to personalized data distributions while collaboratively training the shared projection to improve global generalization. This design is also communication-efficient, as only the shared component is exchanged during communication rounds. Through extensive experiments across eleven datasets, including domain- and label-shift scenarios, we show that pFedMMA achieves state-of-the-art trade-offs between personalization and generalization, outperforming recent federated prompt tuning methods.

Figures (5)

• Figure 1: Few-shot performance across datasets using the ViT-B/16 model. Each radar chart illustrates accuracy (%) for a fixed shot count, with spokes representing the evaluation datasets. Curves correspond to different methods, and values increase outward (20–100%). Accuracy is reported as the harmonic mean (HM) over local, base, and novel classes for each dataset and shot.
• Figure 2: An overview of the pFedMMA framework. Each client independently updates all trainable components of the multi-modal adapters including client-specific up/down projections and the shared projection over local epochs. After local training, only the shared adapter is uploaded and aggregated by the server. This design promotes personalization through local adapters while enabling generalization via a globally shared component.
• Figure 3: Local and harmonic mean (HM) accuracies of various methods across different shot settings.
• Figure 4: Accuracy learning curves of pFedMMA and baselines.
• Figure 5: Accuracy learning curves of pFedMMA and baselines over $10$ clients.