TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models

TL;DR

TOFA addresses the challenge of adapting large Vision-Language Models in federated settings with minimal communication and no additional training. It combines a visual pipeline based on a hierarchical Bayesian model of class prototypes with a textual pipeline that globally aligns LLM-generated prompts, then fuses modalities with a sample-wise adaptive weight to cope with non-IID data. The key contributions are a training-free, one-shot adaptation framework for VLMs, personalized visual prototype learning alongside robust text augmentation, and an adaptive fusion mechanism that balances personalization and robustness. Empirical results across nine datasets demonstrate that TOFA achieves competitive performance with substantially reduced communication and computation compared to training-based federated VLM adaptation methods.

Abstract

Efficient and lightweight adaptation of pre-trained Vision-Language Models (VLMs) to downstream tasks through collaborative interactions between local clients and a central server is a rapidly emerging research topic in federated learning. Existing adaptation algorithms are typically trained iteratively, which incur significant communication costs and increase the susceptibility to potential attacks. Motivated by the one-shot federated training techniques that reduce client-server exchanges to a single round, developing a lightweight one-shot federated VLM adaptation method to alleviate these issues is particularly attractive. However, current one-shot approaches face certain challenges in adapting VLMs within federated settings: (1) insufficient exploitation of the rich multimodal information inherent in VLMs; (2) lack of specialized adaptation strategies to systematically handle the severe data heterogeneity; and (3) requiring additional training resource of clients or server. To bridge these gaps, we propose a novel Training-free One-shot Federated Adaptation framework for VLMs, named TOFA. To fully leverage the generalizable multimodal features in pre-trained VLMs, TOFA employs both visual and textual pipelines to extract task-relevant representations. In the visual pipeline, a hierarchical Bayesian model learns personalized, class-specific prototype distributions. For the textual pipeline, TOFA evaluates and globally aligns the generated local text prompts for robustness. An adaptive weight calibration mechanism is also introduced to combine predictions from both modalities, balancing personalization and robustness to handle data heterogeneity. Our method is training-free, not relying on additional training resources on either the client or server side. Extensive experiments across 9 datasets in various federated settings demonstrate the effectiveness of the proposed TOFA method.

Figures (3)

• Figure 1: Overall Framework of TOFA
• Figure 2: Comparisons on CLIP datasets across varying shot numbers and parameter $\alpha$ in TOFA over 10 clients.
• Figure 3: Results on Inter-Modality Ablation Experiments.

Theorems & Definitions (3)

• Lemma 1
• Theorem 1
• Lemma 2