Efficient Federated Class-Incremental Learning of Pre-Trained Models via Task-agnostic Low-rank Residual Adaptation

TL;DR

This work tackles Federated Class Incremental Learning (FCIL) by addressing catastrophic forgetting and non-IID data across clients. It introduces Fed-TaLoRA, which fine-tunes shared task-agnostic LoRA parameters embedded in transformer layers, while employing a Residual Weight Update (ResWU) mechanism to ensure accurate knowledge consolidation during aggregation. The method achieves strong accuracy gains over state-of-the-art FCIL baselines while substantially reducing computation and communication costs, demonstrated across CIFAR-100, Tiny-ImageNet, and ImageNet-scale benchmarks. The combination of task-agnostic adaptation, post-aggregation calibration, and strategic LoRA placement yields a scalable and efficient approach for resource-constrained FCIL in realistic distributed settings.

Abstract

Federated Parameter-Efficient Fine-Tuning (FedPEFT) reduces communication and computation costs in federated fine-tuning of pre-trained models by updating only a small subset of model parameters. However, existing approaches assume static data distributions, failing to adequately address real-world scenarios where new classes continually emerge, particularly in Federated Class Incremental Learning (FCIL). FCIL faces two key challenges: catastrophic forgetting and performance degradation caused by non-IID data across clients. Unlike current methods that maintain separate task-specific components or suffer from aggregation noise during parameter aggregation, we propose Federated Task-agnostic Low-rank Residual Adaptation (Fed-TaLoRA), a novel parameter-efficient approach for fine-tuning in resource-constrained FCIL scenarios. Specifically, we fine-tune only shared task-agnostic LoRA parameters across sequential tasks, effectively mitigating catastrophic forgetting while enabling efficient knowledge transfer among clients. Based on a theoretical analysis of aggregation, we develop a novel residual weight update mechanism that ensures accurate knowledge consolidation with minimal overhead. Our methodological innovations are attributed to three key strategies: task-agnostic adaptation, post-aggregation model calibration, and strategic placement of LoRA modules. Extensive experiments on multiple benchmark datasets demonstrate that Fed-TaLoRA consistently outperforms state-of-the-art methods in diverse data heterogeneity scenarios while substantially reducing resource requirements.

Figures (12)

• Figure 1: Pipeline of Fed-TaLoRA for FCIL. Clients first receive the global model and fine-tune only their local LoRA parameters embedded in attention layers and FFN (① $\rightarrow$ ②) on own private data containing new classes, then upload these updates (③) for server-side aggregation (④). Additionally, the server computes residual weights to capture cross-client variations, preparing an enhanced model of the next training round.
• Figure 2: Relative final average accuracy ($\%$) compared to Fed-TaLoRA on CIFAR-100.
• Figure 3: The impact of different number of $K$ on CIFAR-100, $\alpha=6$ (left) and $\beta=0.5$ (right).
• Figure 4: An example of the non-IID setting on CIFAR-100 dataset. The value in each rectangle is the number of data samples of a class belonging to a certain client.
• Figure 5: Model performance of LoRA embedded in different blocks for CIFAR-100 dataset.