Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models

Sizai Hou, Songze Li, Baturalp Buyukates

TL;DR

This work proposes SecFPP, a secure FPP protocol harmonizing generalization, personalization, and privacy guarantees, which achieves state-of-the-art accuracy under severe heterogeneity in data distribution.

Abstract

Prompt learning is a crucial technique for adapting pre-trained multimodal language models (MLLMs) to user tasks. Federated prompt personalization (FPP) is further developed to address data heterogeneity and local overfitting, however, it exposes personalized prompts - valuable intellectual assets - to privacy risks like prompt stealing or membership inference attacks. Widely-adopted techniques like differential privacy add noise to prompts, whereas degrading personalization performance. We propose SecFPP, a secure FPP protocol harmonizing generalization, personalization, and privacy guarantees. SecFPP employs hierarchical prompt adaptation with domain-level and class-level components to handle multi-granular data imbalance. For privacy, it uses a novel secret-sharing-based adaptive clustering algorithm for domain-level adaptation while keeping class-level components private. While theoretically and empirically secure, SecFPP achieves state-of-the-art accuracy under severe heterogeneity in data distribution. Extensive experiments show it significantly outperforms both non-private and privacy-preserving baselines, offering a superior privacy-performance trade-off.

Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models

TL;DR

This work proposes SecFPP, a secure FPP protocol harmonizing generalization, personalization, and privacy guarantees, which achieves state-of-the-art accuracy under severe heterogeneity in data distribution.

Abstract

Prompt learning is a crucial technique for adapting pre-trained multimodal language models (MLLMs) to user tasks. Federated prompt personalization (FPP) is further developed to address data heterogeneity and local overfitting, however, it exposes personalized prompts - valuable intellectual assets - to privacy risks like prompt stealing or membership inference attacks. Widely-adopted techniques like differential privacy add noise to prompts, whereas degrading personalization performance. We propose SecFPP, a secure FPP protocol harmonizing generalization, personalization, and privacy guarantees. SecFPP employs hierarchical prompt adaptation with domain-level and class-level components to handle multi-granular data imbalance. For privacy, it uses a novel secret-sharing-based adaptive clustering algorithm for domain-level adaptation while keeping class-level components private. While theoretically and empirically secure, SecFPP achieves state-of-the-art accuracy under severe heterogeneity in data distribution. Extensive experiments show it significantly outperforms both non-private and privacy-preserving baselines, offering a superior privacy-performance trade-off.

Paper Structure

This paper contains 20 sections, 1 theorem, 6 equations, 3 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Federated Prompt Personalization (FPP)
  4. Privacy Preservation in FPP
  5. Problem Overview
  6. Problem Formulation
  7. Threat Model
  8. Solution Overview
  9. Proposed Secure FPP: SecFPP
  10. Cryptographic Building Blocks
  11. SecFPP Workflow
  12. Theoretical Analysis
  13. Experiments
  14. Experimental Settings
  15. FPP Performance Comparisons
  16. ...and 5 more sections

Key Result

Theorem 1

Given a cluster of prompts as normal random vectors by $\overline{\mathbf{P}}_j\sim \mathcal{N}^d(\mu_j,\sigma_j),j\in [n]$, the distance is the $\ell_2$-norm between a prompt $\overline{\mathbf{P}}_i$ and the cluster center $\overline{\mathbf{P}}_{avg}$, i.e., $D^2 = \left \| \overline{\mathbf{P}}_ where $\Gamma, \psi$ represent gamma function and digamma function, respectively; $f_{\overline{\ma

Figures (3)

  • Figure 1: Workflow of SecFPP. On the right, users decompose prompts into global and local components; the global prompt adapts to the dataset domain while the local prompt accommodates the local tasks. On the left, the global prompts are coded by Lagrange coded computation (LCC) and communicated among parties to enable adaptive clustering; then they are aggregated group-wise and distributed to users for the next training round.
  • Figure 2: The mutual information comparisons: distance stands for $MI( \overline{\mathbf{P}}_i; D^2 )$; cluster center is $MI( \overline{\mathbf{P}}_i; \overline{\mathbf{P}}_{avg} )$, which has two cases of the prompt being inside/outside the cluster; self-information is $h(\overline{\mathbf{P}}_i)$. To present the scale, mutual information is plotted in log-scale.
  • Figure 3: The computational cost in seconds per round.

Theorems & Definitions (1)

  • Theorem 1