SDFed: Bridging Local Global Discrepancy via Subspace Refinement and Divergence Control in Federated Prompt Learning
Yicheng Di, Wei Yuan, Tieke He, Zhanjie Zhang, Ao Ma, Yuan Liu, Hongzhi Yin
TL;DR
SDFed tackles the challenge of heterogeneity in privacy-preserving federated adaptation of vision-language models by allowing a fixed-length global prompt while granting each client a variable-length local prompt. Through subspace refinement, it projects local prompts onto a subspace that mitigates conflicts with global directions, and via information retention and divergence control, it preserves key local information while maintaining separability from global representations. The framework achieves strong improvements across single-domain and multi-domain benchmarks, remains robust under model and data heterogeneity, and incurs minimal computational overhead from the refinement step. These results highlight SDFed’s practical potential for personalized, scalable federated prompting in diverse real-world settings.
Abstract
Vision-language pretrained models offer strong transferable representations, yet adapting them in privacy-sensitive multi-party settings is challenging due to the high communication cost of federated optimization and the limited local data on clients. Federated prompt learning mitigates this issue by keeping the VLPM backbone frozen and collaboratively training lightweight prompt parameters. However, existing approaches typically enforce a unified prompt structure and length across clients, which is inadequate under practical client heterogeneity in both data distributions and system resources, and may further introduce conflicts between globally shared and locally optimal knowledge. To address these challenges, we propose \textbf{SDFed}, a heterogeneous federated prompt learning framework that bridges Local-Global Discrepancy via Subspace Refinement and Divergence Control. SDFed maintains a fixed-length global prompt for efficient aggregation while allowing each client to learn a variable-length local prompt to better match its data characteristics and capacity. To mitigate local-global conflicts and facilitate effective knowledge transfer, SDFed introduces a subspace refinement method for local prompts and an information retention and divergence control strategy that preserves key local information while maintaining appropriate separability between global and local representations. Extensive experiments on several datasets demonstrate that SDFed consistently improves performance and robustness in heterogeneous federated settings.