Federated Class-Incremental Learning with Hierarchical Generative Prototypes

TL;DR

This proposal constrains both biases in the last layer by efficiently finetuning a pre-trained backbone using learnable prompts, resulting in clients that produce less biased representations and more biased classifiers.

Abstract

Federated Learning (FL) aims at unburdening the training of deep models by distributing computation across multiple devices (clients) while safeguarding data privacy. On top of that, Federated Continual Learning (FCL) also accounts for data distribution evolving over time, mirroring the dynamic nature of real-world environments. While previous studies have identified Catastrophic Forgetting and Client Drift as primary causes of performance degradation in FCL, we shed light on the importance of Incremental Bias and Federated Bias, which cause models to prioritize classes that are recently introduced or locally predominant, respectively. Our proposal constrains both biases in the last layer by efficiently finetuning a pre-trained backbone using learnable prompts, resulting in clients that produce less biased representations and more biased classifiers. Therefore, instead of solely relying on parameter aggregation, we leverage generative prototypes to effectively balance the predictions of the global model. Our method significantly improves the current State Of The Art, providing an average increase of +7.8% in accuracy.

Figures (6)

• Figure 1: Federated bias. Histogram of the clients' responses computed on the global test set ($\beta = 0.05$). Class indexes have been rearranged for improved visualization (left). The entropy of the response histograms, averaged on all clients, compared with FL performance (right).
• Figure 2: Classifier Rebalancing procedure through hierarchical sampling.
• Figure 3: Prompting vs. fine-tuning. Average pairwise distance of the local prototypes on all clients (left). FL performance before and after Classifier Rebalancing (CR), for $\beta=0.05$ (right).
• Figure A: FAA $[\%]$ in relation with the communication cost [MB] for all tested approaches on Imagenet-R (left) and CIFAR-100 (right).
• Figure B: FAA $[\%]$ in relation with the communication cost [MB] for all tested approaches on CUB-200.