Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

TL;DR

This work develops a theoretical framework for prompt-based federated learning with vision–language foundation models by applying feature learning theory to decompose latent features into task-relevant and task-irrelevant components. It shows that test performance can be characterized by the ratio $\mu/\sigma$ where $\mu$ and $\sigma$ depend on coefficients of global and local prompts, and introduces PromptFolio, a global–local prompt portfolio, with an optimal mixing coefficient $\theta^*$ derived from a mean–variance analogy to portfolio optimization. The authors prove that a well-balanced prompt portfolio can outperform single-prompt baselines, providing an explicit formula for $\theta^*$ and validating the theory with empirical results on multiple datasets under varying heterogeneity and client counts. This approach offers a principled way to balance generalization and personalization in federated vision–language learning and highlights the practical impact of prompt portfolios in realistic, decentralized data settings. Future work could extend the theory to richer text encoders and nonlinearity to better capture deep network dynamics in FL.

Abstract

Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Figures (4)

• Figure 1: The image demonstrates the framework of the PromptFolio algorithm. The algorithm updates the global prompt and local prompt while keeping the weights of the fixed vision-language pretrained model unchanged. Additionally, it aggregates the global prompts from each client. The right side of the image intuitively demonstrates the advantages of global-local cooperation for performance when global and local are treated as two assets.
• Figure 2: The x-axis represents the mixing coefficients, which range from 0 to 1, and the y-axis shows the accuracy of the test set after training. The left figure depicts the result under different data distributions, and the right figure reveals the result under different users.
• Figure 3: The accuracy curve among different shot numbers (left) and different backbones (right).
• Figure 4: The distribution of cosine similarity between different rows of the final projection layer in CLIP CLIP.

Theorems & Definitions (24)

• Lemma 4.1: Feature Representation
• Theorem 4.2: Training Dynamics
• Theorem 4.3: Test Loss
• Theorem 5.1: PromptFolio Advantage
• Theorem 5.2: Optimal Mixing Coefficient
• Lemma E.2: caoBenignOverfittingTwolayer2022
• Lemma E.3
• proof
• Lemma E.4
• proof