Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova
TL;DR
This work tackles the tripartite challenge of personalization, generalization, and privacy in Federated Prompt Learning for multimodal LLMs. It introduces DP-FPL, which decomposes each client’s local prompt into a global component and a low-rank local component $p_i = p_G,i + u_i v_i + r_i$, with the low-rank part updated every round and a residual $r_i$ preserving expressiveness. Privacy is achieved by applying Local DP to the low-rank factors $(u_i, v_i)$ and Global DP to the shared global prompt, using Gaussian noise and careful sensitivity scheduling; this selective noising mitigates utility loss. The approach is validated on several vision-language benchmarks, showing improved personalization-generalization tradeoffs under DP, and displays resilience against Membership Inference Attacks at practical privacy budgets. Overall, DP-FPL offers a practical, privacy-aware pathway to personalized multimodal prompt learning in federated settings, with notable gains in both local and generalized performance.
Abstract
Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.