Personalized Federated Continual Learning via Multi-granularity Prompt

TL;DR

This work tackles Personalized Federated Continual Learning (PFCL), where clients must fuse knowledge across time and space while personalizing models to local data. It introduces FedMGP, a multi-granularity prompting framework that uses coarse-grained global prompts (shared knowledge) and fine-grained local prompts (personalized knowledge), coupled with a selective server-side fusion mechanism to mitigate spatial forgetting. Across CIFAR-100 experiments in both synchronous and asynchronous FCL, FedMGP achieves state-of-the-art performance and demonstrates strong temporal and spatial knowledge retention, validating the effectiveness of the coarse-to-fine cognitive prompting approach. The method also keeps communication and privacy costs low by transmitting only global prompts and their keys, while keeping local prompts and data on-device.

Abstract

Personalized Federated Continual Learning (PFCL) is a new practical scenario that poses greater challenges in sharing and personalizing knowledge. PFCL not only relies on knowledge fusion for server aggregation at the global spatial-temporal perspective but also needs model improvement for each client according to the local requirements. Existing methods, whether in Personalized Federated Learning (PFL) or Federated Continual Learning (FCL), have overlooked the multi-granularity representation of knowledge, which can be utilized to overcome Spatial-Temporal Catastrophic Forgetting (STCF) and adopt generalized knowledge to itself by coarse-to-fine human cognitive mechanisms. Moreover, it allows more effectively to personalized shared knowledge, thus serving its own purpose. To this end, we propose a novel concept called multi-granularity prompt, i.e., coarse-grained global prompt acquired through the common model learning process, and fine-grained local prompt used to personalize the generalized representation. The former focuses on efficiently transferring shared global knowledge without spatial forgetting, and the latter emphasizes specific learning of personalized local knowledge to overcome temporal forgetting. In addition, we design a selective prompt fusion mechanism for aggregating knowledge of global prompts distilled from different clients. By the exclusive fusion of coarse-grained knowledge, we achieve the transmission and refinement of common knowledge among clients, further enhancing the performance of personalization. Extensive experiments demonstrate the effectiveness of the proposed method in addressing STCF as well as improving personalized performance. Our code now is available at https://github.com/SkyOfBeginning/FedMGP.

Figures (6)

• Figure 1: An illustration of constructing a multi-granularity knowledge space in PFCL. By dividing local knowledge into coarse-grained knowledge and fine-grained knowledge, better aggregation of common representations can be achieved on the server side. On the local side, fine-grained knowledge is used to personalize the generalized representation. The two different levels of knowledge can accumulate over time.
• Figure 2: An overview of the proposed FedMGP. Two granularities of prompts are used to capture both temporal-spatial invariant knowledge and specific knowledge. The coarse-grained global prompt is trained through a shared ViT model, acting on the embedding layer. The fine-grained local prompt is built upon the coarse-grained prompt by introducing additional parameters in the MSA layer, enabling the model to better adapt to local data. Moreover, selective prompt fusion is employed to aggregate global prompts on the server side, forming generalized knowledge.
• Figure 3: Ablation Studies using temporal knowledge retention (\ref{['krt']}) spatial knowledge retention (\ref{['krs']}) in two FCL setting. Note that, "Ours-w/oGP" refers to our method without global prompts representing coarse-grained knowledge. "Ours-w/oLP" refers to our method without local prompt to capture client/time relevant knowledge. And "Ours-w/oSPF" is to use FedAvg to aggregate global prompts instead of Selective Prompt Fusion.
• Figure 4: Sensitivity analyses of prompt length and prompt pool size. Left: Global Spatial Knowledge Retention Ratio (%) w.r.t. prompt length $L$ and prompt pool size $M$. Right: Local Spatial Knowledge Retention Ratio (%) w.r.t. prompt length $L$ and prompt pool size $M$.
• Figure 5: Sensitivity analyses of prompt length and prompt pool size. Left: The performance improvement (%) of coarse-grained global prompts after the aggregation of global prompts w.r.t. prompt length $L$ and prompt pool size $M$. Right: The performance improvement (%) of fine-grained local prompts after the aggregation of global prompts w.r.t. prompt length $L$ and prompt pool size $M$.