Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometric Prior-Guided Federated Prompt Calibration

Fei Luo, Ziwei Zhao, Mingxuan Wang, Duoyang Li, Zhe Qian, Jiayi Tuo, Chenyue Zhou, Yanbiao Ma

TL;DR

Federated Prompt Learning suffers from data heterogeneity that biases locally trained prompts. The authors introduce Geometry-Guided Text Prompt Calibration (GGTPC), which privately reconstructs global embedding geometry from local statistics and disseminates a Geometric Prior Calibration Layer (GPCL) to calibrate local embeddings toward global centers, aided by inverse frequency sampling. The approach extends to multi-domain settings with class prototypes, and experiments across label skew, domain skew, and mixed LDS demonstrate robust, plug-in performance improvements over multiple FL baselines. This work shifts the focus from aggregating updates to calibrating training at the feature level, enabling more robust, privacy-preserving FL for vision–language prompt tuning.

Abstract

Federated Prompt Learning (FPL) offers a parameter-efficient solution for collaboratively training large models, but its performance is severely hindered by data heterogeneity, which causes locally trained prompts to become biased. Existing methods, focusing on aggregation or regularization, fail to address this root cause of local training bias. To this end, we propose Geometry-Guided Text Prompt Calibration (GGTPC), a novel framework that directly corrects this bias by providing clients with a global geometric prior. This prior, representing the shape of the global data distribution derived from the covariance matrix, is reconstructed on the server in a privacy-preserving manner. Clients then use a novel Geometry-Prior Calibration Layer (GPCL) to align their local feature distributions with this global prior during training. Extensive experiments show GGTPC's effectiveness. On the label-skewed CIFAR-100 dataset ($β$=0.1), it outperforms the state-of-the-art by 2.15\%. Under extreme skew ($β$=0.01), it improves upon the baseline by 9.17\%. Furthermore, as a plug-and-play module on the domain-skewed Office-Home dataset, it boosts FedAvg's performance by 4.60\%. These results demonstrate that GGTPC effectively mitigates data heterogeneity by correcting the fundamental local training bias, serving as a versatile module to enhance various FL algorithms.

Geometric Prior-Guided Federated Prompt Calibration

TL;DR

Federated Prompt Learning suffers from data heterogeneity that biases locally trained prompts. The authors introduce Geometry-Guided Text Prompt Calibration (GGTPC), which privately reconstructs global embedding geometry from local statistics and disseminates a Geometric Prior Calibration Layer (GPCL) to calibrate local embeddings toward global centers, aided by inverse frequency sampling. The approach extends to multi-domain settings with class prototypes, and experiments across label skew, domain skew, and mixed LDS demonstrate robust, plug-in performance improvements over multiple FL baselines. This work shifts the focus from aggregating updates to calibrating training at the feature level, enabling more robust, privacy-preserving FL for vision–language prompt tuning.

Abstract

Federated Prompt Learning (FPL) offers a parameter-efficient solution for collaboratively training large models, but its performance is severely hindered by data heterogeneity, which causes locally trained prompts to become biased. Existing methods, focusing on aggregation or regularization, fail to address this root cause of local training bias. To this end, we propose Geometry-Guided Text Prompt Calibration (GGTPC), a novel framework that directly corrects this bias by providing clients with a global geometric prior. This prior, representing the shape of the global data distribution derived from the covariance matrix, is reconstructed on the server in a privacy-preserving manner. Clients then use a novel Geometry-Prior Calibration Layer (GPCL) to align their local feature distributions with this global prior during training. Extensive experiments show GGTPC's effectiveness. On the label-skewed CIFAR-100 dataset (=0.1), it outperforms the state-of-the-art by 2.15\%. Under extreme skew (=0.01), it improves upon the baseline by 9.17\%. Furthermore, as a plug-and-play module on the domain-skewed Office-Home dataset, it boosts FedAvg's performance by 4.60\%. These results demonstrate that GGTPC effectively mitigates data heterogeneity by correcting the fundamental local training bias, serving as a versatile module to enhance various FL algorithms.

Paper Structure

This paper contains 24 sections, 7 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Heterogeneity in Federated Learning
  4. Prompt Learning in Vision-Language Models
  5. Federated Prompt Learning
  6. Methodology
  7. Problem Formulation and Motivation
  8. Definition and Acquisition of global Geometric Priors
  9. Definition of Embedding Distribution Geometry
  10. Privacy-Preserving Acquisition of global Geometry
  11. Geometry-Guided Local Calibration
  12. Geometric Prior Calibration Layer (GPCL)
  13. Mitigating Class Imbalance via Inverse Frequency Sampling
  14. Extension to Multi-Domain Scenarios
  15. Overall Algorithm Flow
  16. ...and 9 more sections

Figures (1)

  • Figure 1: PCA visualization of global visual features and text prompt embeddings before/after GGTPC calibration across multiple classes.